Reading and converting a really big dataset! | Just my blog

/dev/random

Reading and converting a really big dataset!

At this moment I am connect to a brand new project, one of the goals of this project is to find out if their is any fraud. For this investigation I received a dump of a SAP-database, with the next data:

  • Transactions payable
  • Authorisation levels
  • Usernames

Only the “accounts payable” are up to 9.200.000 transactions

But the data was not converted in a proper format, so I wrote a tiny PERL script. The code was something like:

And it generates after a while an “Application error”.

During the proces, the memory-usage was rising:

After phoning with a good friend of mine he suggested another method…

While I was running it, the memory usage and CPU-load are stable…

And it works nice and fast

And imho… this is the real cool stuff to do


  • Social

  • By continuing to use the site, you agree to the use of cookies. more information

    The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible and enable advertising to provide you free content. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

    Close