Garbage Collection again

After some experiments, some of my efforts paid off, but not all. Then, I realized it was not just Ruby that used memory. Becaue it's written in RubyCocoa, Cocoa or Objective-C part should use some memory. I believe Objective-C 2.0 has GC, but OS X uses as much memory it has and manage it. I might be wrong, but using more memory itself might not be that bad.

Concord, Cluster, Collocation might be usable with modest amount of memory, but Word Count (n-gram) requires a lot of memory. This is because it creates a huge array (all create arrays, though). I know my current implementation is not ideal, but maybe I have to improve Word Count first. When I was testing the original Ruby scripts, I only used smaller corpora (far less than 100 mil.). Now I need to figure out a way to reduce memory usage, but how? Does anyone have good idea? My implementation is to use hash to count, just as any basic Ruby book shows. But I tweaked it a bit to increase processing speed.

Anyway, this is partly why I put CasualConc can handle 1 mil. corpus at reasonable speed. Well, I need time.

Garbage Collection

I use CasualConc regularly to look up how certain words are used in a context. When I was using it, I realized CasualConc is a memory hog. I knew Word List, espcially when used for n-gram list, needs a lot of memory to process because it keeps counting new ones while it stores counted ones (not exactly, though). I knew Ruby has garbage collection built in, but it seemed like it wasn't working when I wated it to work (maybe because there still was a lot of unused memory). So I decided to force GC to start at some points (GC.start).

But when?

I've been trying several differnt points per each tool and associated method and monitor the differences. But because I've never seriously studied programming (I'm not and have never been in computer science), I don't think I understand how GC works (or in fact, I'm still not sure what exactly OO language entails. If you are breave enough to take a look at the Ruby/RubyCocoa source code of CasualConc, you can see my scripts are not written in Ruby way. I hope I have some time to learn to program a little more seriously someday, but for now, CasualConc works ok (at least for me).

Anyway, I'm not sure if someone ever reads this entry or any entry on this blog, but I'll try to keep my record on this. I want to add some memos on Ruby/RubyCocoa codes on this blog if I can.

PDF to Text converter

In the last post, I wrote I found a way to extract embedded text from PDF files. I wanted to do something with it before I forget, so I wrote a simple utility program in Ruby+RubyCocoa and posted it to CasualConc site. The system requirement is Mac with Leopard. I named it simply PDFtoTextConverter. What it does is open a PDF and show it's embedded text in the text box on the same window. The extracted text can be saved as .txt file. It also has a batch process mode. You can add PDF files to the list and select a folder to save the text as .txt or save .txt file to the same folder where the origial PDF files are stored. If you are interested, please try it. You can go to CasualConc site by following the link on the right.

EDIT: This program is discontinued and integrated into CasualTextractor which is available on the CasualConc Main site under Utility Programs.

PDF

I finally found a way to extract text from text-embedded PDF files in RubyCocoa. I personally don't care about this much, but I guess this might be useful for some people. The problem with handling PDF text is not the extracting part. I mean, the real issues with implementing it to CasualConc are:

1. each line of text is separated by a line feed character LF (\n or \r\n?)
2. page headers/footers, etc. that are not the main text are also included
3. embedded text often includes extra spaces, garbled characters (often with ligatures), etc.

1 is probably the main issue. Currently, the basic unit of analysis in CasualConc is paragraph, which means text separated by LF characters. So it cannot handle text files that separate each line with LF characters such as Brown Corpus files. This require some coding (means not just adding a few lines) and I can't find time to do it now. I'll try to implement this feature in the future, but I don't know when.

2 and 3 cannot be avoided, I guess. So I might try to add a feature to extract text from PDF files within CasualConc, but this also requires certain amount of time.

But at least I know how to extract text from PDF files. So the feature will be included in a future version of CasualConc. If many people are interested, I might prioritize this (but probably won't happen at least until Summer).

Google Search

I've been adding documentation to CasualConc site, although I haven't yet added a download page. Now it has a page for Concordance and Word Cluster along with Basic File Handling.

But now I'm wondering how Google works. I mean the Google Page Creator Help says the page created by it "can be crawled by Google within a few hours of publication". Well, it says "can be", so the actual time might be longer than a few hours. In fact, the CasualConc site was searchable on Google a couple of days after I published it. BUT now it's not on the search result. It disappeared!!

Maybe I should tell my friends to check this first...