Failed to scan very large files

[pombredanne]

While scanning https://github.com/airlift/aircompressor/tree/master/testdata @yashkohli88 reported that there were failures with vert large files in the silesia subdir: mozilla (a tarball), samba (a tarball), webster (a webster dictionary text file from 1913)

We should design a way to handle these properly.

[yashkohli88]

I have tracked down all the details and various scenarios I have tested for this package - clearlydefined/crawler#798