Absolute fastest way to save and read data from text file (or binary file...)

[James Maeding]

Hi there,
I have a prog that looks in a folder, gets a list of files and reads the data from the files. The text files are formatted almost exactly like .ini files.

Right now, the code from my function to get the data looks like this:

(setq FilSys (vlax-create-object "Scripting.FileSystemObject") FilObj (vlax-invoke FilSys "GetFile" FilPth) OpnFil (vlax-invoke FilObj "OpenAsTextStream" 1 0) ) (while (= (vlax-get OpnFil "AtEndOfStream") 0) (setq RetVal (cons (vlax-invoke OpnFil "ReadLine") RetVal))
)

You can see that I am using the scripting FSO to get at the file.

This works dependably and smoothly, but it is too slow. It takes about .03 seconds to read a file. With 300 files, this takes 9 seconds, way too slow.

I am wondering what options I have to speed things up.
Here is what I have thought of so far:
1) Make a DLL with VB that uses windows API functions to generate an array of strings from a text file. Then convert to a list of strings.
2) Convert to using binary files for data, I would need to write DLL's in VB to do this (I think its possible). Convert to list of strings...
3) Convert to using a database, instead of individual text files. This is not desirable.
4) Keep a list of data and only grab data from files that have changed since last "reading".

Anyone out there written an app that desperately needed to read lots of files fast? thanks James Maeding Civil Engineer/Programmer

 

[James Maeding]

Tried it,
average time was 0.030 secs
the average time using FSO was about .036 seconds.

So the read-line method is faster.
Funny thing, in testing I noticed I was actually not using the FSO method origionally. I got my wires crossed on what set of subroutines I was using.

Either way, the .03 seconds is still way too slow.
I need to crank it up about ten times fater.
thanks

"R. Robert Bell"
|>James,
|>
|>I'm curious as what the speed difference is for avoiding FSO. |> |>(setq FileH (open "MyFile.txt" "r")) |>(while (setq TextString (read-line FileH)) |> (setq RetVal (cons TextString RetVal)))
|>(setq FileH (close FileH))

James Maeding
Civil Engineer/Programmer

 

[Devin]

You could try buffering the data perhaps.

Just a thought,

Devin

 

[Kevin Nehls]

How large are the files? 300 is a lot. How many lines on average per file, or just how many lines total?

The only programs I've seen that have really been optimized to read large text files with 1000s or 100,000s of line are those for log file anaylsis and programs are usually written in C, Perl, or even Assembly.

 

[michael puckett]

Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ). You can write an optimized I/O routine that will blow the doors off VB ( approaching optimized C performance ) and yet you can use most of your existing BASIC knowledge.

Hi there,
I have a prog that looks in a folder, gets a list of files and reads the data from the files. The text files are formatted almost exactly like .ini files.

< snippage >

 

[Tony Tanzillo]

Tried it,

average time was 0.030 secs
the average time using FSO was about .036 seconds.

That's interesting, considering that the methods
you're using to measure the execution times are
accurate to about 0.05 seconds.

Try changing your benchmarking test to execute many
times, and only time how long the entire operation
takes, rather than summing or averaging many tiny
execution times, most of which may be flawed.

 

[James Maeding]

that is cool, I was not aware of that complier.
I will pursue that.
thanks

"michael puckett"
|>Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ). |>You can write an optimized I/O routine that will blow the doors off VB ( |>approaching optimized C performance ) and yet you can use most of your |>existing BASIC knowledge. |> |>|>> Hi there, |>> I have a prog that looks in a folder, gets a list of files and reads the |>data from the files. |>> The text files are formatted almost exactly like .ini files. |> |>< snippage > |>

James Maeding
Civil Engineer/Programmer

 

[James Maeding]

I could be fooling myself, but I was using (getvar "date") to compare. I agree that there is error, I was just trying to get an idea of what was taking the longest. No matter how I test it though, its still too slow. I am hoping it is in the file read time rather than the dialog update time. I have no way of changing how fast a dialog updates, the ms flexgrid only goes so fast...
thanks for the comments

"Tony Tanzillo" <tony.tanzillo at caddzone dot com>
|>> Tried it,
|>> average time was 0.030 secs
|>> the average time using FSO was about .036 seconds.
|>
|>That's interesting, considering that the methods
|>you're using to measure the execution times are
|>accurate to about 0.05 seconds.
|>
|>Try changing your benchmarking test to execute many
|>times, and only time how long the entire operation
|>takes, rather than summing or averaging many tiny
|>execution times, most of which may be flawed.

James Maeding
Civil Engineer/Programmer

 

[Byron Blattel]

Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ). You can write an optimized I/O routine that will blow the doors off VB ( approaching optimized C performance ) and yet you can use most of your existing BASIC knowledge.

Michael,

Did you give up on C++?

Just interested

 

[michael puckett]

Nope, but I have over 15 years experience with the compilers written by the cited author, and for those that already know BASIC, it's a seriously practical alternative to VB, esp. where performance is concerned.

I can code BASIC in my sleep; but c++? Man, I have to be WIDE awake. I've been too busy since I finished my university studies to play with c++ in any significant way ( pretty much just weekend recreational reading / experimentation; bummer ); I may have to change careers so I can. It was the very reason I went back to school at mondo expence.

Thanks for remembering / asking; HAGWE

 

[Tony Tanzillo]

Sounds like your problem is not in the LISP that
reads the disk file (because it delegates that to
standard C library functions which are about as
fast as you can get without dropping down to .ASM),
but rather in the part that loads the data into
your UI.

Whether your dialog can be modeless or not is not
really at issue because if the dialog was just
hidden rather than destroyed, you would still have
to account for possible changes to the data that
may occur while it's loaded into a (hidden) dialog,
and refresh it in that case.

It sounds like you need to take a step back and
look at the larger problem (IMHO, the programming
tools that you're using). You need to seriously
consider ditching DCL and anything that resembles
DCL because this is probably one of the worst UI
development environments in existence. I can't
even remember the last time I used it, and I never
will again.

 

[James Maeding]

I'm not sure if you consider ObjectDCL in the same regard as native DCL. I only ose ObjectDCL now though. I agree native DCL is slow and just rather lame. I will take a look at the other processes involved in getting the data to the UI.
Thanks for your insights

"Tony Tanzillo" <tony.tanzillo at caddzone dot com>
|>Sounds like your problem is not in the LISP that
|>reads the disk file (because it delegates that to
|>standard C library functions which are about as
|>fast as you can get without dropping down to .ASM),
|>but rather in the part that loads the data into
|>your UI.
|>
|>Whether your dialog can be modeless or not is not
|>really at issue because if the dialog was just
|>hidden rather than destroyed, you would still have
|>to account for possible changes to the data that
|>may occur while it's loaded into a (hidden) dialog,
|>and refresh it in that case.
|>
|>It sounds like you need to take a step back and
|>look at the larger problem (IMHO, the programming
|>tools that you're using). You need to seriously
|>consider ditching DCL and anything that resembles
|>DCL because this is probably one of the worst UI
|>development environments in existence. I can't
|>even remember the last time I used it, and I never
|>will again.

James Maeding
Civil Engineer/Programmer

 

[James Maeding]

if they are not activex Dll's, can I use them in VisualLisp? I never thought about it before.

"michael puckett"
|>Instead of doing sequential line reads, one technique you might consider is |>opening the file, reading the entire file contents into a single string, |>closing the file, and then continuing on with post processing; parsing etc. |>Depending on the type of data, amount of data and the efficiency of the |>parsing function you use ( power basic has some good core functionality, |>e.g. parse$ ) you may realize dramatic reasults. The PowerBasic compiler |>produces industry standard exe and dlls, though not activex dlls. I've been |>using Bob Zale's compilers since about 1987 -- very fast and tight output. |> |>|>> that is cool, I was not aware of that complier. |>> I will pursue that. |>> thanks |>> |>> "michael puckett" |>> |>Write a DLL using PowerBASIC ( |>http://www.powerbasic.com/products/pbdll32 ). |>> |>You can write an optimized I/O routine that will blow the doors off VB ( |>> |>approaching optimized C performance ) and yet you can use most of your |>> |>existing BASIC knowledge. |>> |> |>> |>|>> |>> Hi there, |>> |>> I have a prog that looks in a folder, gets a list of files and reads |>the |>> |>data from the files. |>> |>> The text files are formatted almost exactly like .ini files. |>> |> |>> |>< snippage > |>> |> |>> |>> James Maeding |>> Civil Engineer/Programmer |>

James Maeding
Civil Engineer/Programmer

 

[James Maeding]

sorry, this thread was posted twice by accident, see the one a few threads up.... James Maeding Civil Engineer/Programmer

 

[James Maeding]

As for C++, I am worried that I will spend a lot of time learning it, then find that it is too dangerous to use. Running down little bugs with Lisp and VB is bad enough. Are my fears founded? I guess C code looks so greek sometimes that the curve seems large. I do know the fundamentals of C though, just never did anything with it.

Byron Blattel <byronatcadwerxdotnet>
|>michael puckett wrote:
|>> Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ). |>> You can write an optimized I/O routine that will blow the doors off VB ( |>> approaching optimized C performance ) and yet you can use most of your |>> existing BASIC knowledge. |> |>Michael, |> |>Did you give up on C++? |> |>Just interested

James Maeding
Civil Engineer/Programmer

 

[Mark Propst]

Excuse me, but what good is a 'DLL using PowerBasic'
going to do in Visual LISP, when Visual LISP cannot
call exported DLL functions to start with?

Hi Tony,
I've been lately reading the posts about invoking dll methods thru lsp. Haven't gotten very far in implementing it yet, still trying to figure it out.
Heres' what I have so far

(defun InvokeDll(dllName dllMethod / ac res res2)
(setq ac (vlax-get-acad-object))
(setq res(vla-GetInterfaceObject ac dllname))
(setq res2 (vlax-invoke-method res dllMethod))
(vlax-release-object res)
(vlax-release-object ac)
res2
)

don't know if it makes any sense or not, but in light of your post I'm wondering if "Exported function" is a special type of function in a dll or what exactly that means? or does this method not work at all to use dlls? I thought there was something wrong with the dll I was trying to test and that was the reason I haven't gotten it to work yet.

appreciate any insight into this you might be willing to share

also interested in *your* answer to the original question of this post cause it's exactly what I've been trying to figure out too, a quicker way to read text files. I wrote a lisp routine that searches all my lisp files for a function name to find out where it's defined. it works but is slow and I've been trying to figure ways to speed it up. Windows search through explorer usually finds the text string faster than my lisp so I'm sure there is some way to do it quicker.

Thanks
Mark

 

[Tony Tanzillo]

Exported functions are functions that a Win32 portable
executable (exe or dll for example) exposes to callers
via one of several standardized binary calling conventions (such as 'stdcall' or 'cdecl'). These calling conventions are an entirely different mechansim from ActiveX.

Delphi, Visual Basic, C/C++, and a number of other langauges can both export functions, and call exported functions using these calling conventions (for example, Windows API functions that are callable from Visual Basic are called this way), but Visual LISP cannot do this. One of several reasons why is because Visual LISP does not support the data types (such as pointers) that can be passed to and returned by exported .DLL functions.

 

[Mark Propst]

Thanks for the info.

so a dll can export functions but it can also have functions that are not exported.? are you saying that powerbasic can only create dlls with exported functions? and that's why it's no use in the way Michael was proposing? looking at the web site for that compiler makes it look pretty impressive as an alternative to vb6 but i don't want to throw money away if it is for some reason limited in it's type of dll it can create.

on an aside if vb can call exported functions, and lisp can call a vb sub, can that be an indirect route around the problem you're pointing out?

and on reading text files i'm wondering if theres' some way to read in binary chunks and somehow convert them to strings afterwards - if that even makes sense - would that be faster? or is reading text files just inherently slow? or is it probably a less than ideal implementation on my part. In my ignorance I would think it should take about a second to read 1000 text files of several hundred lines each. but in practice it takes x minutes.

thanks as always for the education
Mark

 

[Tony Tanzillo]

Thanks for the info.

so a dll can export functions but it can also have functions that are not exported.?

Yes. You don't have to export everything.

are you saying that powerbasic can only create dlls with exported functions?

I'm not saying that, that's what Michael said. If that's the case, then I wouldn't consider it to be a serious development tool, or at least, not as a complement to LISP.

looking at the web site for that compiler makes it look pretty impressive as an alternative to vb6 but i don't want to throw money away if it is for some reason limited in it's type of dll it can create.

Can't advise you on that, but in general, I don't see any reason why someone wanting to code in Basic would use that rather than VB.

on an aside if vb can call exported functions, and lisp can call a vb sub, can that be an indirect route around the problem you're pointing out?

Sorry, now you're not making much sense at all.

and on reading text files i'm wondering if theres' some way to read in binary chunks and somehow convert them to strings afterwards - if that even makes sense - would that be faster?

Not really. That's essentially what the code you're calling is doing now. It just checks each character read until it gets a lf/cr and stops reading. The problem doesn't appear to be reading the file.

or is reading text files just inherently slow? or is it probably a less than ideal implementation on my part.

Can't speculate on code I've never seen.

In my ignorance I would think it should take about a second to read 1000 text files of several hundred lines each. but in practice it takes x minutes.

It's going to take a few seconds, but I don't think it should take even close to 1 minute.

 

[R. Robert Bell]

Thanks. I *knew* I was missing something in Tony's post.

--
R. Robert Bell, MCSE
www.AcadX.com

"Byron Blattel" <byronatcadwerxdotnet> wrote:
| R. Robert Bell wrote:
| > I have nothing but respect for you Tony, but what are you saying? I must be | > mis-reading something in your post. | | Exporting a function is not the same as creating an COM/ActiveX server. For | instance if you look a the the exports for any .arx file you'll always see the | 'acrxEntryPoint' function exported. AutoCAD uses this function to establish a | connection to the arx (dll). COM servers export the following functions:
|
| DllCanUnloadNow
| DllGetClassObject
| DllRegisterServer
| DllUnregisterServer
|
| These functions are used by Windows to register and unregister the type library | and instantiate COM objects among other things. | | The point is that from Visual Lisp you cannot call DllGetClassObject, etc. directly. | | If you're interested in what functions an exe/dll/arx/... exports you can use | depends.exe - | | http://www.dependencywalker.com/ | | It will list all exports (and imports) of a Windows PE file. | | -- | | | ----+---------------------------------------------- | | Byron Blattel | | CADwerx---Applications for AutoCAD | | Autodesk Registered Developer | | email:
| | web site: http://www.cadwerx.net
| |
|