Reading and converting a really big dataset!

At this moment I am connect to a brand new project, one of the goals of this project is to find out if their is any fraud. For this investigation I received a dump of a SAP-database, with the next data:

  • Transactions payable
  • Authorisation levels
  • Usernames

Only the “accounts payable” are up to 9.200.000 transactions

But the data was not converted in a proper format, so I wrote a tiny PERL script. The code was something like:

And it generates after a while an “Application error”.

During the proces, the memory-usage was rising:

After phoning with a good friend of mine he suggested another method…

While I was running it, the memory usage and CPU-load are stable…

And it works nice and fast

And imho… this is the real cool stuff to do