Absolute fastest way to save and read data from text file (or binary file...)

Discussion in 'AutoCAD' started by James Maeding, Jul 17, 2003.

  1. Hi there,
    I have a prog that looks in a folder, gets a list of files and reads the data from the files.
    The text files are formatted almost exactly like .ini files.

    Right now, the code from my function to get the data looks like this:

    (setq FilSys (vlax-create-object "Scripting.FileSystemObject")
    FilObj (vlax-invoke FilSys "GetFile" FilPth)
    OpnFil (vlax-invoke FilObj "OpenAsTextStream" 1 0)
    )
    (while (= (vlax-get OpnFil "AtEndOfStream") 0)
    (setq RetVal (cons (vlax-invoke OpnFil "ReadLine") RetVal))
    )

    You can see that I am using the scripting FSO to get at the file.

    This works dependably and smoothly, but it is too slow.
    It takes about .03 seconds to read a file.
    With 300 files, this takes 9 seconds, way too slow.

    I am wondering what options I have to speed things up.
    Here is what I have thought of so far:
    1) Make a DLL with VB that uses windows API functions to generate an array of strings from a text file. Then convert to
    a list of strings.
    2) Convert to using binary files for data, I would need to write DLL's in VB to do this (I think its possible). Convert
    to list of strings...
    3) Convert to using a database, instead of individual text files. This is not desirable.
    4) Keep a list of data and only grab data from files that have changed since last "reading".

    Anyone out there written an app that desperately needed to read lots of files fast?
    thanks
    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #1
  2. Tried it,
    average time was 0.030 secs
    the average time using FSO was about .036 seconds.

    So the read-line method is faster.
    Funny thing, in testing I noticed I was actually not using the FSO method origionally.
    I got my wires crossed on what set of subroutines I was using.

    Either way, the .03 seconds is still way too slow.
    I need to crank it up about ten times fater.
    thanks

    "R. Robert Bell" <>
    |>James,
    |>
    |>I'm curious as what the speed difference is for avoiding FSO.
    |>
    |>(setq FileH (open "MyFile.txt" "r"))
    |>(while (setq TextString (read-line FileH))
    |> (setq RetVal (cons TextString RetVal)))
    |>(setq FileH (close FileH))

    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #2
  3. James Maeding

    Devin Guest

    You could try buffering the data perhaps.

    Just a thought,

    Devin
     
    Devin, Jul 17, 2003
    #3
  4. James Maeding

    Kevin Nehls Guest

    How large are the files? 300 is a lot. How many lines on average per file,
    or just how many lines total?

    The only programs I've seen that have really been optimized to read large
    text files with 1000s or 100,000s of line are those for log file anaylsis
    and programs are usually written in C, Perl, or even Assembly.
     
    Kevin Nehls, Jul 17, 2003
    #4
  5. Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ).
    You can write an optimized I/O routine that will blow the doors off VB (
    approaching optimized C performance ) and yet you can use most of your
    existing BASIC knowledge.

    < snippage >
     
    michael puckett, Jul 17, 2003
    #5
  6. Tried it,
    That's interesting, considering that the methods
    you're using to measure the execution times are
    accurate to about 0.05 seconds.

    Try changing your benchmarking test to execute many
    times, and only time how long the entire operation
    takes, rather than summing or averaging many tiny
    execution times, most of which may be flawed.
     
    Tony Tanzillo, Jul 17, 2003
    #6
  7. that is cool, I was not aware of that complier.
    I will pursue that.
    thanks

    "michael puckett" <>
    |>Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ).
    |>You can write an optimized I/O routine that will blow the doors off VB (
    |>approaching optimized C performance ) and yet you can use most of your
    |>existing BASIC knowledge.
    |>
    |>|>> Hi there,
    |>> I have a prog that looks in a folder, gets a list of files and reads the
    |>data from the files.
    |>> The text files are formatted almost exactly like .ini files.
    |>
    |>< snippage >
    |>

    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #7
  8. I could be fooling myself, but I was using (getvar "date") to compare.
    I agree that there is error, I was just trying to get an idea of what was taking the longest.
    No matter how I test it though, its still too slow.
    I am hoping it is in the file read time rather than the dialog update time.
    I have no way of changing how fast a dialog updates, the ms flexgrid only goes so fast...
    thanks for the comments

    "Tony Tanzillo" <tony.tanzillo at caddzone dot com>
    |>> Tried it,
    |>> average time was 0.030 secs
    |>> the average time using FSO was about .036 seconds.
    |>
    |>That's interesting, considering that the methods
    |>you're using to measure the execution times are
    |>accurate to about 0.05 seconds.
    |>
    |>Try changing your benchmarking test to execute many
    |>times, and only time how long the entire operation
    |>takes, rather than summing or averaging many tiny
    |>execution times, most of which may be flawed.

    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #8
  9. Michael,

    Did you give up on C++?

    Just interested ;)

    --
    |
    ----+----------------------------------------------
    | Byron Blattel
    | CADwerx---Applications for AutoCAD
    | Autodesk Registered Developer
    | email:
    | web site: http://www.cadwerx.net
    |
     
    Byron Blattel, Jul 17, 2003
    #9
  10. Nope, but I have over 15 years experience with the compilers written by the
    cited author, and for those that already know BASIC, it's a seriously
    practical alternative to VB, esp. where performance is concerned.

    I can code BASIC in my sleep; but c++? Man, I have to be WIDE awake. I've
    been too busy since I finished my university studies to play with c++ in any
    significant way ( pretty much just weekend recreational reading /
    experimentation; bummer ); I may have to change careers so I can. It was the
    very reason I went back to school at mondo expence.

    Thanks for remembering / asking; HAGWE :)
     
    michael puckett, Jul 17, 2003
    #10
  11. Sounds like your problem is not in the LISP that
    reads the disk file (because it delegates that to
    standard C library functions which are about as
    fast as you can get without dropping down to .ASM),
    but rather in the part that loads the data into
    your UI.

    Whether your dialog can be modeless or not is not
    really at issue because if the dialog was just
    hidden rather than destroyed, you would still have
    to account for possible changes to the data that
    may occur while it's loaded into a (hidden) dialog,
    and refresh it in that case.

    It sounds like you need to take a step back and
    look at the larger problem (IMHO, the programming
    tools that you're using). You need to seriously
    consider ditching DCL and anything that resembles
    DCL because this is probably one of the worst UI
    development environments in existence. I can't
    even remember the last time I used it, and I never
    will again.
     
    Tony Tanzillo, Jul 17, 2003
    #11
  12. I'm not sure if you consider ObjectDCL in the same regard as native DCL.
    I only ose ObjectDCL now though. I agree native DCL is slow and just rather lame.
    I will take a look at the other processes involved in getting the data to the UI.
    Thanks for your insights

    "Tony Tanzillo" <tony.tanzillo at caddzone dot com>
    |>Sounds like your problem is not in the LISP that
    |>reads the disk file (because it delegates that to
    |>standard C library functions which are about as
    |>fast as you can get without dropping down to .ASM),
    |>but rather in the part that loads the data into
    |>your UI.
    |>
    |>Whether your dialog can be modeless or not is not
    |>really at issue because if the dialog was just
    |>hidden rather than destroyed, you would still have
    |>to account for possible changes to the data that
    |>may occur while it's loaded into a (hidden) dialog,
    |>and refresh it in that case.
    |>
    |>It sounds like you need to take a step back and
    |>look at the larger problem (IMHO, the programming
    |>tools that you're using). You need to seriously
    |>consider ditching DCL and anything that resembles
    |>DCL because this is probably one of the worst UI
    |>development environments in existence. I can't
    |>even remember the last time I used it, and I never
    |>will again.

    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #12
  13. if they are not activex Dll's, can I use them in VisualLisp? I never thought about it before.

    "michael puckett" <>
    |>Instead of doing sequential line reads, one technique you might consider is
    |>opening the file, reading the entire file contents into a single string,
    |>closing the file, and then continuing on with post processing; parsing etc.
    |>Depending on the type of data, amount of data and the efficiency of the
    |>parsing function you use ( power basic has some good core functionality,
    |>e.g. parse$() ) you may realize dramatic reasults. The PowerBasic compiler
    |>produces industry standard exe and dlls, though not activex dlls. I've been
    |>using Bob Zale's compilers since about 1987 -- very fast and tight output.
    |>
    |>|>> that is cool, I was not aware of that complier.
    |>> I will pursue that.
    |>> thanks
    |>>
    |>> "michael puckett" <>
    |>> |>Write a DLL using PowerBASIC (
    |>http://www.powerbasic.com/products/pbdll32 ).
    |>> |>You can write an optimized I/O routine that will blow the doors off VB (
    |>> |>approaching optimized C performance ) and yet you can use most of your
    |>> |>existing BASIC knowledge.
    |>> |>
    |>> |>|>> |>> Hi there,
    |>> |>> I have a prog that looks in a folder, gets a list of files and reads
    |>the
    |>> |>data from the files.
    |>> |>> The text files are formatted almost exactly like .ini files.
    |>> |>
    |>> |>< snippage >
    |>> |>
    |>>
    |>> James Maeding
    |>> Civil Engineer/Programmer
    |>

    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #13
  14. sorry, this thread was posted twice by accident, see the one a few threads up....
    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #14
  15. As for C++, I am worried that I will spend a lot of time learning it, then find that it is too dangerous to use.
    Running down little bugs with Lisp and VB is bad enough.
    Are my fears founded?
    I guess C code looks so greek sometimes that the curve seems large.
    I do know the fundamentals of C though, just never did anything with it.

    Byron Blattel <byronatcadwerxdotnet>
    |>michael puckett wrote:
    |>> Write a DLL using PowerBASIC ( http://www.powerbasic.com/products/pbdll32 ).
    |>> You can write an optimized I/O routine that will blow the doors off VB (
    |>> approaching optimized C performance ) and yet you can use most of your
    |>> existing BASIC knowledge.
    |>
    |>Michael,
    |>
    |>Did you give up on C++?
    |>
    |>Just interested ;)

    James Maeding
    Civil Engineer/Programmer
     
    James Maeding, Jul 17, 2003
    #15
  16. James Maeding

    Mark Propst Guest

    Hi Tony,
    I've been lately reading the posts about invoking dll methods thru lsp.
    Haven't gotten very far in implementing it yet, still trying to figure it
    out.
    Heres' what I have so far

    (defun InvokeDll(dllName dllMethod / ac res res2)
    (setq ac (vlax-get-acad-object))
    (setq res(vla-GetInterfaceObject ac dllname))
    (setq res2 (vlax-invoke-method res dllMethod))
    (vlax-release-object res)
    (vlax-release-object ac)
    res2
    )

    don't know if it makes any sense or not, but in light of your post I'm
    wondering if "Exported function" is a special type of function in a dll or
    what exactly that means?
    or does this method not work at all to use dlls? I thought there was
    something wrong with the dll I was trying to test and that was the reason I
    haven't gotten it to work yet.

    appreciate any insight into this you might be willing to share

    also interested in *your* answer to the original question of this post cause
    it's exactly what I've been trying to figure out too, a quicker way to read
    text files. I wrote a lisp routine that searches all my lisp files for a
    function name to find out where it's defined. it works but is slow and I've
    been trying to figure ways to speed it up. Windows search through explorer
    usually finds the text string faster than my lisp so I'm sure there is some
    way to do it quicker.

    Thanks
    Mark
     
    Mark Propst, Jul 18, 2003
    #16
  17. Exported functions are functions that a Win32 portable
    executable (exe or dll for example) exposes to callers
    via one of several standardized binary calling conventions
    (such as 'stdcall' or 'cdecl'). These calling conventions
    are an entirely different mechansim from ActiveX.

    Delphi, Visual Basic, C/C++, and a number of other langauges
    can both export functions, and call exported functions using
    these calling conventions (for example, Windows API functions
    that are callable from Visual Basic are called this way), but
    Visual LISP cannot do this. One of several reasons why is
    because Visual LISP does not support the data types (such as
    pointers) that can be passed to and returned by exported .DLL
    functions.
     
    Tony Tanzillo, Jul 18, 2003
    #17
  18. James Maeding

    Mark Propst Guest

    Thanks for the info.

    so a dll can export functions but it can also have functions that are not
    exported.?
    are you saying that powerbasic can only create dlls with exported functions?
    and that's why it's no use in the way Michael was proposing?
    looking at the web site for that compiler makes it look pretty impressive as
    an alternative to vb6 but i don't want to throw money away if it is for some
    reason limited in it's type of dll it can create.

    on an aside if vb can call exported functions, and lisp can call a vb sub,
    can that be an indirect route around the problem you're pointing out?

    and on reading text files i'm wondering if theres' some way to read in
    binary chunks and somehow convert them to strings afterwards - if that even
    makes sense - would that be faster?
    or is reading text files just inherently slow? or is it probably a less
    than ideal implementation on my part.
    In my ignorance I would think it should take about a second to read 1000
    text files of several hundred lines each.
    but in practice it takes x minutes.

    thanks as always for the education
    Mark
     
    Mark Propst, Jul 18, 2003
    #18
  19. Yes. You don't have to export everything.
    I'm not saying that, that's what Michael said. If that's the case,
    then I wouldn't consider it to be a serious development tool, or
    at least, not as a complement to LISP.
    Can't advise you on that, but in general, I don't see any
    reason why someone wanting to code in Basic would use that
    rather than VB.
    Sorry, now you're not making much sense at all.
    Not really. That's essentially what the code you're calling
    is doing now. It just checks each character read until it
    gets a lf/cr and stops reading. The problem doesn't appear to
    be reading the file.
    Can't speculate on code I've never seen.
    It's going to take a few seconds, but I don't think it should take
    even close to 1 minute.
     
    Tony Tanzillo, Jul 18, 2003
    #19
  20. Thanks. I *knew* I was missing something in Tony's post.

    --
    R. Robert Bell, MCSE
    www.AcadX.com


    "Byron Blattel" <byronatcadwerxdotnet> wrote in message
    | R. Robert Bell wrote:
    | > I have nothing but respect for you Tony, but what are you saying? I must
    be
    | > mis-reading something in your post.
    |
    | Exporting a function is not the same as creating an COM/ActiveX server.
    For
    | instance if you look a the the exports for any .arx file you'll always see
    the
    | 'acrxEntryPoint' function exported. AutoCAD uses this function to
    establish a
    | connection to the arx (dll). COM servers export the following functions:
    |
    | DllCanUnloadNow
    | DllGetClassObject
    | DllRegisterServer
    | DllUnregisterServer
    |
    | These functions are used by Windows to register and unregister the type
    library
    | and instantiate COM objects among other things.
    |
    | The point is that from Visual Lisp you cannot call DllGetClassObject, etc.
    directly.
    |
    | If you're interested in what functions an exe/dll/arx/... exports you can
    use
    | depends.exe -
    |
    | http://www.dependencywalker.com/
    |
    | It will list all exports (and imports) of a Windows PE file.
    |
    | --
    | |
    | ----+----------------------------------------------
    | | Byron Blattel
    | | CADwerx---Applications for AutoCAD
    | | Autodesk Registered Developer
    | | email:
    | | web site: http://www.cadwerx.net
    | |
    |
     
    R. Robert Bell, Jul 18, 2003
    #20
Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.