CasualConc - a concordancer for Mac OS X

CasualConc beta bug fix

2011-12-04T09:50:00.001+09:00

I got a bug report, so I fixed it.

The problem was in File Info. When exporting a Word Freq Info result, low frequency counts of individual files were sometime omitted. This was because the cells with no numbers were not skipped. So when it reached the number of types in a file, CasualConc stopped handling the data for that file for exporting. Internally, the frequency counts were stored (you could see them on the window), so I made it sure that CasualConc handle all the data properly.

If you find any problem, please let me know.

CasualTranscriber alpha

2011-11-27T03:05:00.001+09:00

Since the current version of CasualTranscriber is quite buggy because of the programming language I use, I decided to rewrite it in another language. Now, the new version has most of the basic functions for transcribing, so I decided to release it as an alpha version. It does not have all the functions that the current version has and is more likely to have bugs, but it is, in a sense, more stable (at least in my environment).

If you are using CasualTranscriber, especially on Lion, please try it and let me know what you think.

Lion compatibility and ...

2011-09-30T23:16:00.000+09:00

It's been a couple of months after Lion was released. Today, I made some changes to application so that they run on Lion (at least). I don't have time to test all the features, so if you use them on Lion, please let me know if they work fine or have some problems.

And finally, I decided to drop the Leopard support. They might run, but I will not test the compatibility any more (I finally upgraded my Leopard machine to Snow Leopard).

If you really need to use them on a Leopard machine, let me know.

Also, I've added a few minor features to CasualConc. It now has a function to test your regular expressions. Go to Main Menu -> Window -> Regex Test Panel. This is an experimental feature, so let me know what you think.

Another minor feature is a support of multi-line regular expression search. You can enable this on the Preference window.

I've also started to rewrite CasualTranscriber. I labeled it as alpha, but most of the basic features of the current version are added. It only runs on 64-bit Mac with Snow Leopard or Lion. If you have chance to try it, let me know.

CasualConc on Lion update

2011-07-28T00:26:00.000+09:00

I figured out how to include sqlite3-ruby in CasualConc, so I made changes to the latest beta build and labeled it as CasualConc beta for Lion. It is available on the CasualConc download page. Please try it and let me know if it doesn't work (or it does).

CasualConc on Lion

2011-07-25T00:13:00.003+09:00

I haven't been able to check the compatibility personally, but I got a couple of reports that CasualConc 1.0.x does not run on Lion.

UPDATE: As you can see above (at least for now), I figured out how to include sqlite3-ruby, so you don't have to follow this process unless the beta for Lion doesn't run.

To use CasualConc beta on Lion, you need to install 'sqlite3-ruby' and run a beta build 2011/07/21 or later.
1. open Mac App Store.app and download/install Xcode (free) [you need Mac App Store account]
2. open Terminal.app
3. type sudo gem install sqlite3-ruby --version "= 1.2.5" and hit the enter key (and enter the password of your account on Mac)

I will check this as soon as I can access Lion (hopefully within a couple of weeks), but I heard the beta ran after installing sqlite3-ruby.

I will also check other apps and see if I can make them run on Lion (if they don't).

Cumulative bug fixes and feature additions to CasualConc beta

2011-03-10T13:41:00.000+09:00

I haven't posted anything here for a while because I've been busy with developing a new version of CasualConc and preparing for the classes I start teaching next month.

In the last few months, I added a few new features and fixed bugs (mostly the ones I introduced when I added these new features).

Bug fixes

General
- searching multiple words/phrases in Word (wildcard) mode using a slash (/) matched words/phrases that included searched words/phrases

Concord
- keyword was not correctly colored in the context view when searched in the database mode with File as Scope of Context
- keyword was not colored correctly in the context view when used in the database mode (introduced with the above change)
- saved results did not display correctly

Feature additions

Concord
- added positions in a file as an sort option when File is selected as Scope of Context

File info
- added standardized TTR (TTR in every 1000 words)

Concordance Plot
- you can reflect the changes you made to Concord results
- able to export selected/all plots as a single PDF file.

If you find any other bugs or want to see some other features, please let me know. I'll try to fix bugs as soon as possible and I will try to add requested features as much as possible (unless they are technically too difficult).

Bug fixes to CasualConc Beta 1.8

2010-11-04T17:03:00.000+09:00

I found a few bugs related to Corpus/Database file handling in the Advanced mode. Also, a serious bug was reported, so I fixed them and uploaded the latest beta to the site. If you have downloaded beta 1.8, please visit the site and download the latest beta. The latest beta is also version 1.8, but if you could run and check About CasualConc, it should say Version 1.8 (20101104).

If you still cannot run this version or find another bug, please report it to me. You can email me or post comment on the blog or send message on Twitter.

A minor update to CasualConc Beta

2010-11-01T12:59:00.002+09:00

I added a few features (some new and some enhanced) to CasualConc Beta. I also enabled an experimental gapped n-gram list feature.

Documentation is not updated, so you might need to figure out how to use some of the features.

General
- Spelling variation feature. You can register spelling variations (i.e. analyze-analyse) and use them in Concord/Cluster/Collocation searches as well as in Word List.
- You can assign different corpus/database file to the left and right tables in Cluster/Word Count in Advanced Corpus Handling Mode.

Concord
- Sorting now has 4th elements.

Cluster
- longer cluster search (up to 8 words)

Word List- Gapped n-gram (3-5 gram)

Concordance Plot
- You can export selected concordance plot as JPEG files (individually) or print them (= export as PDF).

The spelling variation feature is to accommodate word search/word list creation in languages that have spelling variations (i.e. American/British English). Once you create a spelling variation list, you can use information on it when searching a word/creating a word list. There is one problem with this feature. Unless your corpus and spelling variation list are not pos-tagged, CasualConc cannot distinguish the same spelling of different word classes (i.e. analyses [v] vs analyses [n]). A sample list of spelling variations is included in the disk image. You can import it to CasualConc to see how this feature works. You might want to create your own list (there might be some errors in my list and it is far from complete).

For corpus handling, I added a feature to assign different corpus/database file to the left/right tables in Cluster and Word Count. This is available in Advanced Corpus Handling Mode. If you have more than one corpus or database file registered on the table and check more than one corpora or database files, you can select one corpus/database file or All for each of the two tables in Cluster and Word Count. You need to do this in File view.

In Concord, you can select 4th sorting position. I don't know how useful this feature is, but I wanted this from time to time, so I added it. Also related to Concord is Concordance Plot export. Now you can export Concordance Plots. There are two types. One is to export selected plots individually as JPEG files. You can select the ones you want to export on the plots. Another is to print the selected plots. Thanks to OS X's "Save as PDF" feature, you can save the plots as a single PDF file. Now that you can export plots, you might want to change the size of plots, I guess. So I added a feature to change the hight and width of plot boxes. You can set them in Preferences -> Others.

In Cluster, you can create a Cluster list of up to 8 words (= 7 + search word).

In Word List, you can search gapped n-grams (2-5 grams). This feature was already introduced as an experimental feature. I enhanced it a bit and enabled it. I'm thinking about enhancing this feature, but I decided to release it before that hoping to get some feedback (to decide how to enhance this feature).

There might be bugs related to these new features. Any feedback and/or bug report is welcome.

CasualConc bug fix

2010-10-01T02:11:00.000+09:00

I found a bug on CasualConc, so I fixed it.

Concord
- The number of files count was incorrect when used with the context word search option ON

Also on CasualConc beta, a little more serious bug was found in the database mode.

General
- Results of Cluster, Collocation, and Word Count were incorrect when the context word search option was ON and a context word(s) was specified in Concord. This happened ONLY in the database mode. There was a bug in creating an SQL query.

If you find any other bugs, please let me know.

CasualConc beta another minor update

2010-09-28T10:03:00.000+09:00

I found a bug in the process to add files to an existing database file, so I fixed it and made some minor changes to corpus file handling in advanced mode.

File (in Advanced Corpus Handling mode)
- you can now add files to an existing database file (this feature was broken)
- you can delete multiple files from a corpus/database file
- you can find duplicate files in a corpus/database file -> identify files with the same path
- you have an option to move a database file to Trash when you delete one from the table

The bug I found was in the process of adding files to an existing database file. This worked long time ago and I somehow broke it when I made changes in the process. But since I haven't got any bug report, I guess no one really uses this function.

And when I fixed this, I also made some changes in corpus handling in the advanced corpus handling mode. Now you can select multiple files on the corpus/database content table (lower left). This means you can delete multiple files at one go. Related to this, I added a feature to identify duplicate files in the selected corpus/database file. If CasualConc find duplicate files, the ones added first (older ones) are selected. Once files are selected, you can delete them at once.

Also you can now move a database file to Trash when you remove it from the database list table.

Finally, I added alert messages related to the corpus handling. When a corpus/database file was not selected or files are not added to a table to process, CasualConc simply ignored button clicking. Now, it gives you a message explaining why CasualConc doesn't process your request in the file view (in most of the cases).

As always, if you find any bug, please let me know.

CasualConc beta minor update

2010-09-08T23:26:00.000+09:00

I haven't updated this blog for a long time. Since I last posted here, I made a few changes to CasualConc beta.

- Exporting Concord result as RTF

You can now export Concord results as a RTF document. With this format, you can choose to keep coloring of sort words as well as font style of context words.

- Searching Cluster/Collocation from other tools

Though it was not documented on the site, you could run Concord search from the Cluster, Collocation, and Word Count tables. Now you can search Cluster and Collocation from the Word Count table.

If you are using CasualConc beta or want to try the beta, please go to the CasualConc site and download the latest beta.

I'd appreciate any feedback on the beta.

CasualTextractor documentation and a few minor bug fixes

2010-06-16T10:23:00.001+09:00

I finally updated documentation of CasualTextractor. I added quite a lot of features to the PDF mode while ago, but haven't had time to document them. Now the new page is lined from the CasualTextractor page.

Also I fixed a few minor bugs I found while I was documenting it and made a few minor changes to it while I was documenting the new features. The current version is 0.7.1. If you use CasualTextractor or and wanted to know how to use it in the PDF mode, please check the site.

A CasualConc bug fix

2010-04-30T13:25:00.001+09:00

I got a bug report and fixed it (thank you, Adriano).

Bug fix
- crashed when searching with wildcard in the Database mode.

The report was about Concord, but I think it also happened in Cluster and Collocation/Cooccurrences.

This is not a problem on the beta version. I fixed this on the beta when I found this bug, but forgot to apply it to the working version.

You can download CasualConc from the CasualConc site.

If you find any other bugs on the working version (1.0.2), please send a bug report to me. Currently, I'm only using the beta version, so I will only fix bugs which are in common with both the working version and the beta version unless I receive a report.

The current status of CasualConc beta

2010-04-21T21:43:00.002+09:00

The current status of CasualConc beta - experimental features 2

2010-04-10T11:36:00.000+09:00

This post will be the last of this series of posts 'The current status of CasualConc beta'. The last feature is also a experimental one. It's a gap n-gram list creation (for a lack of better word).

What this does is simple, you can create a n-gram list with one of the words in n-gram (3-5) as a gap or wildcard or whatever you call it. In the experimental beta version, when you select 3-gram, 4-gram, or 5-gram in Word Count, a check box appears.

Check this box and click Count.

Because this process can take a long time and needs a lot of memory if your corpus size is big, a warning message appears. When I tried this with a corpus size of 500,000, this process took almost 10 minutes.

If you are brave enough, this is what you get. The corpus I used is Inaugural Addresses of the Presidents of the United States corpus, prepared by Prof. Tabata at Osaka University for a workshop I attended. As you can see, the gap is represented by * and the words that appear in that slot is in the Gap words column with frequency information.

You can select one of the line and see the entire list of gap words on a table.

Select a line and right click the table.

A panel with a table appears with the list.

You can copy the list and paste it on other applications. In this case all the gap words appeared on the Word Count table, but this is basically designed to see all the words when they are not displayed.

You can see all of them on the table.

OK, that's it. I think I covered almost all the new and enhanced features for the next version (current beta) of CasualConc. The current beta has all these new features except for the last two experimental features. You can download the beta version from Download CasualConc Beta page (Japanese page).

If you are interested in the experimental beta build, please contact me directly at casualconc (at) gmail.com.

Since this is a beta version, CasualConc can be unstable and might have bugs related to the changes I have made. If you ever try this beta version, I'd appreciate your feedback/bug reports.

The current status of CasualConc beta - experimental features

2010-04-10T04:35:00.001+09:00

In this post (and possibly the next) I will present two new experimental features of CasualConc. These two features are still too experimental to be included in a beta on the site. So if you get interested after reading this post, please contact me directly. The email address is on the CasualConc main site.

The first one is visualization of collocations and the second is n-gram search with a gap. I will explain what these mean.

The visualization of collocations is an idea proposed by Prof. Tabata at Osaka University, Japan. This is simply a realization of his idea (not mine). Now, let me show you the current implementation.

To use this function, you need a word list and collocation with the same corpus. So run Word Count and Collocation. The Visual (name is tentative) button is on the Collocation tool. Click it to display the Visualizer panel.

The settings of the Visualizer as follows:

The top one is which statistics to use for visualization. The choices are shown below.

Then select the span/position of collocated words. The upper choice is a position from L5 to R5. If you select R1, only the information on the R1 position (frequency) is used to calculate a selected statistic. If you enable Span, the calculation will be based on the tally of frequency up to the selected position. For example, if you select R3 and check Span, the information in R1, R2, and R3 positions will be used to calculate statistics. The lower choice is a span to the left and right of the keyword. You can select from L1~R1 to L5~R5.

Take xxx words means the first xxx words on the Collocation table will be used for visualization. So if you sort the results on the Collocation table, the words taken from the list will be affected.

Now let's see what this does.

With this setting, the result will look like this. Larger the number, bigger the font size.

This simply reflect the frequency of words in the L1 position. Our (80) is the most frequent and American (40) follows. But the picture is quite different with MI (Mutual information).

You can also incorporate frequency information with other statistic result. If you enable Include Freq Info, the frequency information will be added with a gray scale.

The result will look like this:

If you click Stats button, you can see the actual numbers on a table. You can sort by alphabetical order of words or stats values.

If you check Ignore zero occurrence, words with zero frequency will be removed from the display.

If you choose Log-Likelihood, because of its values, higher values can go extreme, so you can convert LL value with log(10). To enable this, click Convert LL val to log.

With the original LL values, the image will look like this:

With the conversion, it will look like this:

The choice is up to you.

The most experimental part is visualization with 4 statistics. By clicking Use Multiple info, you can incorporate three additional statistics values.

The current implementation is highly experimental and not tuned to display most effective color scheme, but basic idea is that value of each statistic can be assigned to one of RGB. Higher the value, lower the color value. So if the value for a certain word is high in all three, the font color should be closer to black. If in the above example, Log-Likelihood value is very low and z-score value and Log-log value are very high, the font color should be close to red. These are primary colors, so 100% on all of them means white and 0% on all of them means black.

When the above three statistics are applied with MI as the primary statistic, the image will look something like this:

Blueish or Greenish font colors mean relative values of z-score and Log-log are low compared to a relative value of Log-Likelihood. But the actual values of each statistic can vary a lot, the displayed color scheme may not reflect a true relationships among statistics. I need to figure out the way to visualize the optimum relationships among statistics values. If you have any suggestion, I'd most appreciate it.

Finally, the statistic values of all four indicators can be checked on the stats value table.

You can sort the items by clicking the header of columns.

That's about it. As I mentioned at the beginning, this feature is not available on the current beta. If you'd like to try this, please contact me directly. My email address is on the CauslConc sites (the links are on the right side column of this blog).

The current status of CasualConc beta - Interface

2010-04-10T02:35:00.000+09:00

The most salient difference in the new version for Japanese users is the interface. If you use CasualConc in Japanese language environment, interface items and messages will be displayed in Japanese. Here is the example from Concord.

Messages are also in Japanese (unless I forgot to change them).

Of course, Preferences are also in Japanese.

When this new version is out of beta, I will include help files in Japanese (the current beta does not have help files included).

OK, this is the end of new feature show case. These should be available in the most up-to-date beta.

For the next couple of posts, I will show you some experimental features. Those are not enabled in the beta on the site, but if you are interested in testing the features, please contact me directly.

The current status of CasualConc beta - Corpus File Information

2010-04-10T02:18:00.001+09:00

Perhaps Corpus File Information gets the most enhancement.

In the current working version, Corpus File Info only creates basic file information and the number of n letter words, which is not very interesting. In the new version, you can do a lot more with Corpus File Info.

Corpus File Info now has three modes.

Basic Info is what the current working version does. Basic frequency stats with frequencies of n-letter words. Word Freq Info is to create a frequency matrix (frequencies of specified words in each file). TF-IDF is a measure of prominence or or importance of words in a file. For more information read this Wikipedia entry. To run TF-IDF analysis, you need to run Word Count or import a word list with the information regarding how many files in a provided corpus a certain word appears.

Let me start with Word Freq Info. If you select Word Freq Info, the following items appear on the window.

First you select how you create a list of word you count in the selected corpus.

You can select one of the three sources.

Once you select the source click Import button. You can limit the number of words you import in Preferences -> File Info. If you uncheck this, CasualConc tries to import all the word available in the source.

If you select From Word Count, the words on the left table of Word Count from the first one to the number specified in the Limit or all if not. The order of words to be imported is whatever order on the Word Count. So if you sort the list by alphabets, the imported list is in that order.

If you select From File and click Import, you can import a word list, you will be prompted to specify the format of the word. You can only import a plain text file with a certain format (CSV or tab-delimited). You can specify how many columns from the left or rows from the beginning. You can click Check button to check what will be imported.

If you select From Import Panel and click Import, the import panel appears.

You can directly enter or copy/paste a list (one item/word per line). Once you finish creating a list, click Read button.

You can check what are imported, by clicking Check button on the main window.

You can sort words in alphabetical order by clicking the column header. The last column is the original order of the imported words. You can delete words from the list if you want.

Once you are sure about the list, you can specify the range of words to count. If you want to count only the first 20, you enter 1 and 20 in the boxes.

Then click Get File Info. With the default settings, the result will look like this. The header is the word that were counted and the numbers are the frequencies of the word in each file (and Total).

If you check Normalize word freq in Preferences -> File Info, you can convert the frequency to percent or per xxx words.

If you select %, the result will look like this.

If enable Sort frequency list of each file by word frequency, you can sort the result for each file by the order of frequency index.

If you create a list with percent and sort the result by frequency, you will get a result like this:

The process to import word list is the same for TF-IDF. But to run TF-IDF, you need to have a word list on Word Count and the list should have the information of the number of files a certain word appear in the corpus. If you run Word Count, this information is on the table. Here is the result of the default setting. .00 means the word appears on all the files in the corpus.

You can select how to sort the results in Preferences -> File Info.

If you select Sum of all files, the TF-IDF values for a word will be added up and the sorting is based on the sum of the value of TF-IDF on each file.

If you select Each file, the sorting will be done for each file based on the TF-IDF values.

Personally, I've never used this for my research, but it seems to be a well-known indicator in text mining.

In any case, if you want calculate TF-IDF values for all the words, but only display a limited number of words, uncheck Limit the number of words to import to and import all the words from Word List. Then set Limit result table columns to a reasonable number.

I once tried this with no limits on both with a corpus of about one hundred thousand tokens and the corpus had several thousand unique words. This means the table had several thousand columns. When I tried to scroll the table, even scrolling only one row took several minutes. So don't try it!

The current status of CasualConc beta - Word Count

2010-04-10T00:40:00.002+09:00

There is only one new feature on Word Count (although creating n-gram list should be a little faster).

You can now specify what to count in Regular Expression. Go to Preferences -> Others and check Regular Expression Mode for Word Count. You can count whatever you want as long as you can write a regular expression for it.

Now, the functionality of tag modes will be presented here.

Word Count in Tag(s) mode with Separate Word and Tag in WC NOT checked, the result will look like this:

If Separate Word and Tag in WC IS checked, the result will look like this:

If Tag Only mode is selected, the result will look like this:

N-gram in Tag(s) mode with Separate Word and Tag in WC NOT checked, the result will look like this (4-gram):

If Separate Word and Tag in WC IS checked, the result will look like this:

If Tag Only mode is selected, the result will look like this:

That's it for Word Count.

The current status of CasualConc beta - Cluster/Collocation/Cooccurrence

2010-04-10T00:24:00.003+09:00

Only one minor change is made to Cluster.

When you select Left Only in Span, the result will be aligned to the left.

If you select Tag(s) mode in search word, the result of searching 'jj nn1' (adjective + singular noun) will look like this:

If Suppress tags in context is on, the result will look like this.

And Tag Only search will return results like this:

Now the Collocation. In the current working version, the frequency of the total includes the frequency of the keyword position. But in the new version, the total is only the total of word in the context. So the keyword no longer comes on top of the list.

Another change is variable span. Now you can set the span of the context words and do it separately for the left and right up to 5 words.

The result will look like this:

Collocation Stats calculations should reflect this to some extent.

Now with Tag(s) selected, the result will look like this:

If Treat Keywords As One Word is checked, the result will look like this:

If Suppress tags in context is on, the results with or without Treat Keywords As One Word checked will look like this:

NOT Checked

Checked

Finally, Tag Only search will return the result like this:

The next is Cooccurrence.

The new feature of Cooccurrence is sorting. You can now sort words in each position based on collocation statistics. To use this feature, you need to run Word Count first.

In the normal frequency order, the result looks like this:

With MI (Mutual Information) sort, the result will look like this:

With Tag(s) mode, the result will look like this:

Now, you can export Cooccurrence result with frequency information. Check Include frequency info in the Cooccurrence export in Preference -> Others.

The exported csv file with freq info will look like this on Excel.

Now with tag handling on Cooccurrence. With Tag Only mode, the result will look like this:

These are the new features of Cluster/Collocation/Cooccurrence.

The current status of CasualConc beta - Concord

2010-04-09T23:33:00.002+09:00

Now the Concord.

It now has a few minor feature additions. First is independent left/right span. You can set the span of context texts on the right and left independently.

So if you want to see more on the right of the keyword, you can set the span on the left to a smaller number of characters and set the right one to a larger number. So the result would look like this:

The second is wider context word span. In the current working version, you can only search context words up to 5 words to the right and left and you can only search single words. With the new version, you can search up to 10 word on the right and left of the keyword and you can search a phrase. To enable wider context words, go to Preferences -> Concord and check Wide Context.

Now you can select up to 10 words on the right or left.

On the preferences, you can select Words Only or Words and Phrases for context word search.

This means you can search like this:

You can also specify words you don't want to see in the context. This means if the specified word(s) appear in the specified span, that line will be excluded from the result. To enable this, check Exclude right next to the Context Word Mode setting (see above). A text box and span settings for Excluding words appear on the main window. Check the box next to Exclude and search the keyword.

The result would look like this:

In this example, a word 'the' should not appear on the left and the right of the keyword with in 10 words (if this is working correctly).

Another feature is font selection. With the current working version, you can only select Courier or Courier New (or appropriate font might be chosen for non-alphabetic languages). Now you can select any font on your system. To do this, go to Preferences -> Concord and click Manage in the Display setting.

A font panel appears. You can select a font from Pop-up menu and click Add to add to the list. To remove a font from the list, just select a font and click Remove.

You can now select this added font.

Another feature is related to copy/paste or export kwic results. If you want to paste copied lines on Pages or MSWord or any application that accepts Rich Text Format text, you can paste the lines with the keyword in bold. Check Keep text style when copying the results.

If you paste the text on a document, the copied text preserves the font, font size and bold text on the keywords.

You can also insert a tab before and after the keywords. This works with a plain text format and rich text format. Check Insert Tab before/after Keyword.

The pasted text should look like these:

Plain Text

Rich Text

This setting (tab) should work when you export the results.

Another feature is editing the original/database text. First, you need to enable this in Preferences. You can allow editing of the original file and editing of database files.

For Allow editing of original file, you can select Only in File Mode or In Both Modes.

If allow editing of original file is on, after you run Concord and select a line on the table, right-click (or go Main Menu -> Text Data) the table.

If you select Open Displayed Text, a editor panel appears.

You can edit the original text file and save the changes. This comes in handy when you find errors in the kwic result. This has a basic tagging helper function. You specify tags on Tag Panel and then open Tag Drawer and click the number next to the tag text to insert a tag.

If you select Open Displayed Text with Application, the file opens with a specified application (the default is TextEdit).

In Database Mode, you can edit a database entry.

Select Edit Database Entry of the selected Line.

You can make changes and update the entry or delete the entry.

If the original text file is in the same directory when you created the database, you can select Open Displayed Text with Application to open the file with a specified application.

And in both modes, if you select Show File of the selected Line in Finder, the original text file will be displayed in Finder (of course if the file is still in the same directory in Database mode).

Another minor addition is searching text in the Context view. You could just copy and paste to search any word you find in the Context view, but you can directly do it in the Context view. Just select a word (or phrase) in the Context view and right-click to select Search in Concord.

Finally, I will briefly go over how tag-handling works.

In Preferences, you can select Tag(s). Tag Only mode does not work in Concord because looking at kwic lines of just tags does not make much sense.

Once you select the Tag(s) mode, select a tag type.

In Tag(s) mode, you can run kwic just by typing tags. This example search 'jj nn' (adjective + noun) combination.

You can suppress tags in the context. In Preferences, check Suppress tags in context.

Then the results should look like this:

But this suppressing tags in context has one issue. When you click a kwic result line, the displayed context text and keyword coloring in the context text is not correct. I will fix this if I can find a good way, but until then, this remains as is.

The current status of CasualConc beta - General/Global part 2

2010-04-09T17:25:00.000+09:00

Lemma and Keyword grouping. The changes are mostly with how to deal with lemma/keyword group lists. In the current working version, you can just select a lemma or keyword group file. In the next version, you can manage them on CasualConc.

To use lemma/keyword group function, go to Preferences -> Lemma (this process is the same).

The difference is you select a group you manage on lemma/keyword group panel. If you check Apply Lemmatization to Search Word, searching a word returns all the words with the same stem.

To manage lemma list, go to Main Menu -> Window -> Lemma List Panel. There are separate panel for Lemma and Keyword Group.

A panel appears.

On the left table, you manage groups just like stop word/skip character list. You can enter a lemma set or remove/modify them. You can also import an existing lemma file such as e-lemma file. The lemma/keyword group lists can be exported in the same format. You can create multiple lists and apply any of them to your analysis.

In file handling, you can specify any file extensions by checking Allow Other File Types on Preferences -> Files. Or you can allow any file types (even the files with no extension) by checking Any File Types. If there is any readable text on files, CasualConc tries to read it.

Ignore tag/section has a little improvement. You can select either xml/html type tags or under score type tags (<*> or ~_*).

You can also specify any string to ignore in the analysis. To enable this, check Ignore Specified Strings. You can specify whatever strings you want in regular expression. The blank column with a check box determines whether the entry is used or not, C is case sensitivity (check to make the regular expression case sensitive) and M is to apply regular expression across line feed characters (\r\n,\n,\r). You can edit regular expressions in the text box or on the table.

Now, I will focus on tools from the next post.

The current status of CasualConc beta - General/Global

2010-04-09T16:28:00.003+09:00

I start with some features that are common across tools.

First is Search Word Choice.

Experimental tag search options are added. This feature is only supported in European Language A/B mode and works in Concord, Cluster, Collocation, and Word Count. What this does in each tool will be explained (if I have time) in the post for each tool. But generally, if you select Tag(s) in Concord, Cluster, Collocation, you can search tagged corpus just by tag. In Word Count, you can choose to display words and tags in separate columns. For example, if your corpus is POS-tagged and jj is used for adjectives and nn is used for nouns, searching 'jj nn' returns all the sequence of 'jj nn' in your corpus, such as 'beautiful_jj day_nn'. Tag Only does not work in Concord, but in Cluster, Collocation, and Word Count, you can create lists only with tags.

If you select Regular Expression, you can set case sensitivity.

Another new feature is setting character replacement. In the current official version (1.0.2), some of the characters (mostly symbols, non alphabet characters) are automatically replaced. For example, smart quotes (“”) which usually used in word processing applications are treated as a regular character (like alphabets) because of their assignments in Unicode (double byte?). Also if you copy/paste or convert text from MSWord or read MSWord documents directly, there would be many multi-byte characters. In the new version (or beta), you can specify (in other words, you need to specify) replacement.

To enable this feature, in Preferences -> General, check Replace Characters.

Then, click Show List to open Replace Characters panel. Click Add button to add a new entry and enter character directly on the table. To apply any of the replacement pairs, check the box on the left before you run the analysis. This feature is not fully functional, so if you enter a weird character on the table, it might not accept it (this happened when I enter a garbage character [a half of the Unicode code of a character]).

Next is Include Words. You can specify some sequence of character to be treated as a word. In the current official version, you can specify this on the Preferences, but they are always applied to any files. With this new function, you can create different groups of characters. To enable this, check Include Items on 'Include Words' list on the Preferences -> General.

Clicking the Include Words List button will display a panel, but this is integrated with Stop Word/Skip Character list, so I will explain this with them below.

Also you can specify any character to be treated as a part of a word (if this works properly). Check Other characters to be included and specify any character you want. If you check Includes word initial, a word that starts with the specified character should be treated as a word with them (I'm not sure how well this works, though).

Stop Word/Skip Character function. This is a totally new function. You can create and manage stop word/skip character. The latter is for multi-byte character language, such as Japanese, to ignore punctuation characters. In these languages, punctuation characters are also multi-byte, so they are treated as a regular character (like alphabets). So this function is to avoid it.

To use this function, go to Main Menu -> Window -> Stop Word/Skip Character List Panel.

A panel appears. On the left, you manage groups. Fist, enter a group name and click Add button to create a new group. Then select a group on the left table. You can delete or rename a group. On the right, you have three choices: Stop Words, Skip Characters, and Include Words. Select an appropriate tab and enter a word/character, and click Add button. You can remove any word by clicking Remove button. Also you can import/export the list. This accepts a plain text file and you can select an encoding. The format if one word/character per line. Exported text will be in the same format (one word/character per line).

You can set if you want to apply stop words on particular tools. Go to Preferences -> Others and check the tool to apply stop word deletion.

The language/group choice and stop/skip word application as well as a current search word mode are now displayed on the main window. SK is stop word is enabled, and SK is Skip Characters is enabled. You can switch Search Word mode and stop word/skip character group on the main window.

Well, this post is getting long, so I stop here.

The current status of CasualConc beta

2010-04-09T15:27:00.000+09:00

It's been a while since my last post... I've been busy with annotating and analyzing my own corpus for my research. While I was working on my own corpus, I've made a lot of changes to CasualTagger and some on CasualConc. The analysis of my corpus text is almost done and now I need to concentrate on writing. But before that, I decided to make note of the changed I've made on CasualConc beta. I'm not sure if anyone is following this blog, so this is mainly for my own purpose.

I will make multiple posts today (and maybe tomorrow) to cover most of the changes and how-tos for new features, so that those who is (bravely) using the beta version will know what some of the new buttons, choices, etc. do.

I will also try to present some experimental features that will appear in the coming beta. These features are still underdevelopment, but if you are interested, I'm more than happy to share it with you. Just email me (address is on the main site) so that I can send you the most up-to-date beta.

Also if you ever try the beta version, please send me any feedback or post comments on this blog. I can read Japanese and English, so both are OK.

Now let me start with general or global feature in the next post.

CasualTagger 0.8

2009-12-18T02:36:00.001+09:00

I'm not sure if there is anyone who have even tried CasualTagger, but because I've been using it to tag my own corpus, I've been adding features to it. I don't have time to work on the documentation with screenshots, so I'll describe new features and how to use them (to some extent) on this blog. But because I don't remember all the changes I've made since the last update, I only mention more significant ones here.

General feature
- memo panel

You can keep notes of what you are doing and CasualTagger keeps you memo. The format of this might change in the future, though.

In Editor mode
- coloring of specified strings in kwic context (up to 2)
- specifying left/right kwic context span separately
- adding xml type tag in addition to pos-tags (with shortcuts)
- more versatile tag coloring
- search word/coloring string history

The coloring is simple. You just specify word/phrase/whatever to add colors in kwic context. You can specify if you add colors to left/right/both context. The mode of search word (wildcard/character/regular expressions) applies to context string coloring.

For kwic span, you can now specify it for left and right context separately.

The xml tagging feature is added. So now you can add two different types of tag formats (one for pos-tags and the other for xml type tags). Both can be done with shortcuts. For example, you can add pos-tags in _XXX format and at the same time, you can work on xml type tags ~.

Tag coloring is more versatile now. You can specify different types of tags including xml type tags.

Search word and context coloring strings has history features just like search word/context word history in CasualConc.

New modes
- Item Counter
- Custom File Info

Item Counter is simply to count occurrences of strings in your corpus that match a regular expression. To use this, first add files to the file list table on the left. Then open Option panel (Menu -> Window -> Counter Option Panel). You can specify the end of the file information part (just like you can in CasualConc). You can also specify any strings to ignore in counting (with regular expression). If the files have any string that match the specified regular expression, they will be deleted before CasualTagger counts what you want to count. You can have any number of items on a table and check the ones you want to apply. You can also specify them for each table. If you use () to back reference, only those in the brackets will appear on the table.

Custom File Info is basically multiple Item Counter. To use this, add files to the file table on the left. Then click Settings button on the top right corner. You can specify end of the file info part and any string to ignore in all the counts.

On the setting table, add items to count. Label is a label for table columns. Check "U" if you want to count only unique occurrences. "C" is case sensitivity for regular expression. Check "M" to allow multiple line matches. Then specify items to count and items to ignore in regular expressions. You can specify multiple items for Items to ignore. Just use a comma [,] to separate regular expressions. You can export/import that list for later use. Drag and drop to change the order.

Once you set everything, close the setting window and click Run. You can export the result as a tab-delimited text (in UTF-8) to open in Excel/Numbers or any spreadsheet application.

You can use Item Counter to check how your regular expressions work in Custom File Info, though it's not perfect (it doesn't have ignore case/multiple line for ignore items in Item Counter.

Anyway, I don't know if there's anyone to try CasualTagger, but if you are interested, you can download it from the CasualConc site. If you ever use it, please let me know what you think.

CasualConc update and CasualTranscriber

2009-11-08T01:53:00.002+09:00

Well, I stated here that there would be no more new feature in version 1.0, but I changed my mind... This is due to a couple of reasons. One is I found a couple of bugs and I had to fixed them. Another is adding them was not so time consuming. I added to the current beta first and mostly simply copy/paste the items and scripts to version 1.0. Especially one of the new features is what I wanted to include in version 1.0, but hadn't figured out how.

Anyway, the current version is 1.0.2 and here's the list of bug fixes and added features.

CasualConc Version 1.0.2

Bug fix
- In Cluster, the same cluster was counted twice if a search word/phrase appears twice in a cluster (such as 'is that is')
- Related to the above one: in Cluster, if a search word appears twice in a cluster, only one word was colored
- In Cluster and Collocation with Lemma search is on, not all words with the same lemma appear in the list (right most column)

The bug in Cluster was a little serious. In the original script, clusters were collected on every search word. This means that if there is a sequence 'is that is' and you search 'is', 'is that is' was counted twice (once with the first 'is' and then with the second 'is' in the cluster). Now, when this happens duplicates are eliminated, so 'is that is' is counted only once. And because it was assumed the search word appears only once in a cluster, only one of them was colored. Now if there are two occurrence of the search word in a cluster, both should be colored.

The bug with Lemma is a minor one, which just means I don't expect many people use this function. When you run Cluster or Collocation with the Lemma feature on, CasualConc shows the frequencies of words in the same lemma (or clusters with them) on the very right column. But when Lemma contains only one word, it didn't display correctly. Now all the words should be displayed.

New features
- Concordance Plot -- you need to set 'Scope of Context' to 'File' and check 'Create Concordance Plot'
- Search word/context search word history

Concordance Plot shows where in a file the search word appears. The plots are generated when you run Concord with 'Scope of Context' set as 'File' and 'Create Concordance Plot' is checked in Preferences.

Another new feature is search word/context search word history. CasualConc remembers the words you searched in Concord/Cluster/Collocation and you can select one from the pull-down menu. You can set how many search/context search words to remember in Preferences (in General [Search Word] and Concord [Context Word]).

I hope these new features didn't introduce new bugs.

I also updated CasualTranscriber, a transcription helper application. Now it should run on Mac OS X 10.6 Snow Leopard. I also fixed some bugs and added new features. It should be a little more stable.

The new/enhanced features are insert tag function and much more powerful regular expression search/replace.

To use tag function, go to Menu -> Window -> Tag Panel. Then enter tags under Tag. To insert a tag, type Command + Option + (the number on the left). So to insert the tag in 1, type Command + Option + 1. If Tag is laugh, the tag will be inserted as . If any text is selected on the text view, it appears between the tags. If you check the box and enter options, they appear like attributes in XML. Options will be divided by a comma (,) so if you enter option1,option2 in the Options box, the inserted tag will be . The selected text appears between tags.

If you don't want to type the combination, you can click a button to insert a tag. On the main window, you will see a button on the top right corner (the one to show tool bar on any Cocoa application). Click it to show an icon like a gear. Clicking the gear icon shows a drawer on the right. Click a button next to the tag you want to insert. You can change the tag on the drawer (the change will be reflected on the panel). But if you want to change the options, you need to do that on the panel.

I fixed some bugs, but I can't tell which ones. This is because so many things broke in Snow Leopard and I couldn't tell which bugs were in the previous version and which ones were due to change in the OS.

Anyway, if you try either of them and find any bug, please let me know.

Another CasualConc bug fix

2009-11-01T10:48:00.001+09:00

In the development of the next version, I found a bug which affects the version 1.0 and fixed it.

Bug
- the context text (in context view) is not properly displayed when the search word appears in the first paragraph of a file.

This only affected if you use Database mode and the searched word appears in the very first paragraph of a file (original file, not the database file).

The development of the next version is slow. Because I started to change the fundamentals, only a part of functions work now (file management and a part of Concord). Anyway, here's the list of tentative new/revised features (might not show up in the next version).

- stop word list
- skip character list
- experimental pos tag search/count (only in European Language modes)

If you have any suggestions, please let me know. I'll see if I can include them in the next version.

CasualConc minor bug fix

2009-10-25T02:15:00.000+09:00

I found a bug in CasualConc when I was working on the beta, which has the same script. It's a minor one (a feature I believe not a lot of people use).

Bug fix
- crashed when creating a database file in Advanced Corpora Database mode using tag deletion.

If you are one of the rare people who uses this function, please download the latest version.

CasualConc 1.0 and more...

2009-10-19T02:15:00.000+09:00

This post will a long one.

I've decided to make the latest build of CasualConc a version 1.0 mostly because I didn't get feedback/bug report (well, I guess not a lot of people ever tried the latest beta or people don't bother to report any bugs...). Anyway, I made a few changes to it and here's the list. Bug fixes are very minor

Casualconc version 1.0

Bug Fix
- timer for File Info now working
- move tables in Cluster moves everything including span and type
- word list import now functions

Enhancement
- creating File Info table is much faster
- progress bar in File mode progresses based on the number of files processed
- now including help files (the same content as you find on the site; English only)

From now on, this version is only maintained if I ever get bug report. I've already started adding new features and I'm planning to make more fundamental changes to it. I'll release it as version 1.x some time in the future, but no time frame. If you have any feature request, please send them to me. I'll try to add them once I add what I want, though whether I can add what you want depend on my time and scripting skills. I might release it as beta (or alpha) if people are interested even before it becomes stable. If you are, please let me know.

In addition to this, I've also updated some of other applications. I don't know if anyone ever uses them, but I really started using some of them personally, so I've been trying to include what I want. Well, the reasons I started to write the applications vary. Some were upon requests and some were just experimental (to try what I learned in scripting). Now, I want to make them more like real applications.

Anyway, the updated applications and the details of updates are below. I don't think people are interested, but these are for the record, so I can keep track of what I've done.

CasualMecab

- Aozora Bunko Kanji substitute handling
- experimental Word List function using Mecab output
- Snow Leopard Support (separate build)

Aozora Bunko Kanji substitute handling is to pick the Kanji substitute, such as ※［＃「てへん＋劣」、第3水準1-84-77］ and replace with real Kanji's (now this is possible with Unicode characters). Word List function uses Mecab format output and create word list with any of the info available (not just with the word on the text). You can create a word list of base form and part-of-speech combination, etc. Snow Leopard Support is just a work around. If you use Snow Leopard, you need to download the one for it.

CasualTagger

- support rbtagger if installed
- better regex search/replace
- delete punctuation tags
- ignore header section (info part?)
- skip bracketed tags from tagging
- progress bar in batch process
- run tagger in editor mode

CasualTagger now support rbtagger. You can find more information about rbtagger here. rbtagger is a tagger based on Eric Brill's tagger by Todd A. Fisher. You need to install it by yourself, but it's very simple. Just type sudo gem install rbtagger in Terminal.app. It still has some issues, but it's good to have alternatives.

Regex replace was supported before, but now you can use it for search. Delete punctuation tags delete tags put on punctuation characters (not words). Ignore header section is for my own purpose. Some of my corpus files have header section ~ and I don't want to add tags to the text in this section. So now CasualTagger can ignore this part (keep the original text). Skip bracketed tags are to ignore section tags I have on some files, such as ~, etc. And the progress bar is added to batch processing. Finally, you can apply tags (engtagger/rbtagger) on a single file in Editor mode.

CasualTextractor

- PDF mode
- search in PDF
- enlarge/reduce in PDF
- go to selected text (from PDF to extracted text)
- delete selected text (on PDF from extracted text)
- split word search and replace (PDF artifacts)
- replace character list

- Web mode
- open web files (html/htm/webarchive)
- clear open page/file
- web history
- source view

- Document
- split word search (for PDF)
- replace character list

Overall
- open recent files
- regular expression search
- simple tagging support
- text format options (replace certain text/characters)

I've made so many changes, so I make notes on some of them.

In PDF mode, with delete selected text on PDF, you select text on PDF view and delete the section. The text will be deleted from the text view and the text on PDF will be struck through. This is handy if you want to delete header or footer on the PDF text. Split word search is to find words split when PDF file was created, such as interest- ing due to line break.

In Web mode, you can open web files (not just drag&drop) and clear the page to allow you drag&drop another file. Web history is what you usually see in a browser, though it's limited. You can see the source of the page and make changes to it (you can see the result of the change).

Overall, it has regex search/replace.

The information on the site is still based on the previous beta. I probably won't update it until I'm certain the features are set. But if you try and wonder how a function works, please feel free to contact me. Also any bug report is welcome.

CasualPConc 0.7 and a new application

2009-09-28T22:12:00.000+09:00

I made a few changes and fixed a few bugs on CasualPConc, a simple parallel concordancer for Mac OS X. It is a little more stable now. I also worked on the documentation. Now it covers most of the features.

A new application is based on CasualPConc. When I first released CasualPConc, someone asked if I would make it to handle more than two corpora. This is kind of my answer to that. CasualMultiPConc has limited features, but it can handle up to 5 parallel corpora.

This new application simply does kwic concordance of up to 5 parallel corpora. The future of this application is up to users (if there's any). I don't have any experience in parallel corpus analysis and I don't need to use it right now, so I'm not even sure how well this works. I only use small corpora to test it. If you are interested in testing it, any feedback is welcome. This one is also for Mac OS X 10.5 Leopard or later, though I only tested it on 10.6 Snow Leopard.

This application and CasualPConc are only on English site (Main Site link on the right). Both of them are under Other Applications.

CasualConc updated, but still not 1.0

2009-09-23T01:42:00.001+09:00

I fixed a couple of bugs and added a few features. Yes, I wrote I wouldn't spend time on East Asian language support, but I somehow figured out how to handle coloring in Concord with 2-byte character modes (Japanese (plain) and Japanese (wakachi)). Well, it's in the middle of 5-day weekend in Japan...

Bug fixes
- crashes in Word Count when n-gram list is created in File Mode.
- File Information treated word lengths in bytes not in characters

Feature Improvements
- added two new Word Count sort options: Word Length and Reverse Word Length
- added Character as a search word choice
- full regular expression search
- a progress bar is added at the bottom of the main window, though it doesn't indicate the progress (it shows CasualConc is processing your request)
- much better East Asian Language support (Japanese (plain) and Japanese (wakachi))

Along with better East Asian Language support, I made a few other changes. Two new Word Count sort options goes with File Information. Now you can get information about the number of words with certain characters/letters. So this new feature is to check which ones are the longest/shortest words.

An addition of a new search word mode is to search for the characters used for wildcard search. Now you can search * ? ! and other non-word characters.

The change in regular expression search is that before this change, all the regular expressions are word level. In other words, the actual regular expression processed inside CasualConc was inside the \b~\b (word boundaries). Now this limitation is lift. So if you want the same results as before, simply put \b in front and after the regular expression.

The progress bar added to the main window only shows CasualConc is processing. It doesn't show how much processing has done.

East Asian Language support is much better now. Context word coloring is added and now you can use the database mode. But because of the nature of texts (no spaces between words), some of the functions behave differently. More detailed information about East Asian Language support is documented on the site (only on the English site at this moment).

Now I will finally focus on bug fixes and minor changes. I won't make any more major changes before 1.0, or so I think...

The documentations on CasualConc and CasualTagger are updated. They should reflect the latest versions.

I'd appreciate if you could report any bugs as soon as you find them.

CasualConc update

2009-09-20T03:05:00.000+09:00

Just because I wanted to get file information, I added it to CasualConc.

New feature
- File Information

It is very basic and only returns type, token, type-token ratio and number of n-letter words.

Improved
- Fisher's Exact Test calculation speed

I also rewrote the algorithm for Fisher's Exact test and it is much much faster, though I'm sure no one has used this since I added it last time. But I haven't fully tested this, so I'd appreciate if anyone can test its accuracy.

Now the version is 0.9.9.9. Well, it's almost 1. If I can't find any other bugs when I use it in the next couple of weeks or I don't get any bug report, I'll simply make it 1.0. I'm sure it still has bugs even if I make it 1.0, but that's the nature of computer programs.

So, my decision for now is that I don't spend any more time on East Asian Language support because I haven't heard from anyone who uses CasualConc with East Asian languages. I personally don't use CasualConc even with Japanese, so I don't see any necessity. I'll work on this later if I have time, but I'll focus more on other programs once this hits 1.0. So if you use this with East Asian languages and would like to have better support for East Asian languages, please let me know.

If I think of other nice features, I'll probably add it to other programs and see if it works with CasualConc before I added them to it. In fact, I added File Info to CasualTagger and it wasn't too complicated, so I decided to add it to CasualConc (well, I wanted this feature, but haven't tried to write scripts).

Anyway, if you use CasualConc or other programs, please, please let me know what you think.

CasualConc and CasualTagger updates

2009-09-17T02:15:00.001+09:00

Tonight, I uploaded newer versions of CasualConc and CasualTagger.

CasualConc's update is minor.

New features:
- Fisher's Exact Test in Collocation Stats Calculators (experimental)
- Calculator for 2x2 contingency table

The first one is added upon request by someone who kindly checked accuracies of stats calculation. Thank you, Sebastian! It looks like most of the stats are reasonably accurate. Anyway, because the calculation of the Fisher's p-value is CPU intensive (esp. with large N), I made it as an option. To include Fisher's p-value, go to Preferences -> Other and check 'Include Fisher's Exact Test'. I haven't got report of the accuracy of this, so I'm not sure how well it works. If anyone can test it, I'd appreciate it. The contingency table calculator is based on the same formulas with other stats calculation. It returns Log-Likelihood, chi-square and Fisher's Exact Test (optional). I hope this is useful for someone.

The update to CasualTagger is also feature enhancements.

Enhanced features:
- word count now works with untagged text ('None' is added to options)
- kwic search for specified word(s)/phrase(s) is available (it was only possible from a word list)
- simple sort in kwic
- word count and kwic with multiple files (optional)
- editor now accepts text files encoded other than UTF-8 (set in Preferences)
- ignore specified tags or file information (by specifying an end marker/tag) in word count and kwic

Because I made so many changes, there might be many bugs. Now I'm trying to tag my own corpus, so I've been making changes to suit to my needs. I'll make more changes as I need them, but if you ever try CasualTagger and have nice ideas, please let me know. I'll try to include them if they are not too complicated or they look useful for my work. Also I'll try to update the documentation.

Also I checked CasualMecab on Snow Leopard, but it doesn't work. I installed MeCab and MeCab-Ruby on Snow Leopard and it works fine from Ruby scripts (I tested the exact same script). But somehow MeCab-Ruby doesn't work in an application. I'll try to fix it if I can find any solution.

CasualPConc update

2009-09-13T01:36:00.000+09:00

Somehow, CasualPConc didn't run well in Snow Leopard. This could be because Ruby in Snow Leopard is updated to 1.8.7 from 1.8.6 and this change might have caused errors.

Anyway, I fixed some major bugs. There might be some other bugs which are caused by the same source (related to Array Controller). I also updated the how-to on the site.

If you find any bugs, please let me know.

CasualConc bug fix

2009-09-05T00:38:00.002+09:00

I found a bug in Word Count when I was cleaning up the codes for it.

Bug fix
- crashed when creating n-gram list in the Database mode.

This bug was introduced when I added a warning message for missing files in the last update.

As for the clean-up, the problem was in Cluster and Word Count, separate codes were written for each table (right and left). This was because of my lack of scripting skill (I still don't have much). I couldn't think of a good way to identify which button was clicked and process them accordingly. If you know how Cocoa works, this should be obvious, but when I started this project, I had no experience in Cocoa.

The new version is 0.9.9.7. It's almost 1.0, so I'll try to wrap up to make it 1.0 soon. This means no more major feature before 1.0 and I'll focus on bug fixes. But unless I hear a lot from users whether it is mostly bug free or still has many bugs, I'm not confident enough to make it out of beta, though beta simply means (to me) it's not tested enough. Computer programs will never be bug-free.

Anyway, if you find any bug, esp. in Word Count and Cluster, please let me know.

Snow Leopard

2009-08-29T00:21:00.000+09:00

Apple's new OS, Mac OS 10.6 Snow Leopard was released yesterday. I installed it on my Mac mini and did some tests with CasualConc. All the basic functions seem to work fine. I haven't checked every single feature, but I don't expect to see any serious issues with this upgrade.

Yet, if you find any broken feature or any other bugs, please let me know.

CasualConc minor update

2009-08-27T01:14:00.000+09:00

I uploaded a newer version of CasualConc last week. I haven't had time to work on this for a while, but I got one bug report and one minor feature request, so I fixed the bug and added the feature. The new version is 0.9.9.6.

Bug fix
- crashed when a lemma file is selected and the Lemma mode is on but CasualConc can't find the selected lemma file (it was moved, deleted, etc.).

New feature
- file name (FN) can be selected in sorting Concord results

If you find any other bugs, send it to me at casualconc (at) gmail.com. If the bug is very serious, I'll try to fix it as soon as possible.

CasualConc update

2009-05-08T20:17:00.000+09:00

I got a couple of bug reports and a feature request, so I fixed them and add the feature. I also found some other bugs related the reported one and fixed them too. In addition to them, I made one minor change.

Bug fixes
- crash in Concord with Scope of Context set as Sentence in the Database mode.
- corrupted export CSV files from Collocation
- crash in saving Collocation table

Addition
- Reverse Alphabetical sorting in Word Count

What this does (if it works) is to sort words in alphabetical order, but from the last letter to the first letter. So in the normal alphabetical order, a, an, that, the, this are ordered in this order, but in Reverse Alphabetical order, the order will be a, the, an, this, that.

Change
- settings of minimum frequency in Cluster, Collocation/Coocurrence, and Word Count are moved to Preferences

Now you can't set Min Freq. for each table in Cluster and Word Count, but CasualConc remembers the Min Freq. for each tool.

I also got a request to support exporting results in Excel format. I've been experimenting this and this is on the to-do list (I can't tell you when I will add this because I need to figure out how to implement this first). This would probably require you to install a Ruby Module in Terminal (a single line of command). Is there any other people who are interested?

CasualConc quick bug fix

2009-03-30T19:56:00.000+09:00

I just found a bug in CasualConc. When it opens kwic result in a new window, it crashes. If you use this function, please download version 0.9.9.3 from the site. I think this was introduced when I made a few changes last time.

If you find any other bugs, please let me know. If they are minor and easy to fix, I'll try to fix them in a day or two.

CasualPConc more updates

2009-03-30T19:08:00.000+09:00

Today, I learned at least one person in the world knows CasualPConc exists other than myself. I'm really glad that.

I added a few more features to CasualPConc today. Now almost all the functions I can think of and I wanted to add are there. I might add a function to export results if anyone is interested. Or if anyone has a good idea, I might consider that. But from now on, I'll focus on bug fixing and documentation. I'll update the CasualPConc page on the CasualConc main site in the coming weeks.

I got one request to make CasualPConc be able to handle more than two parallel corpora. But I think it's hard to add that function to CasualPConc. It would probably be easier to write a new program based on CasualPConc. I might work on this once I finalize CasualPConc and if I have time to focus on its development.

Anyway, if you happen to be reading this blog and are interested, please try it ang give me some feedback. Using basic functions should not be difficult. Or you can wait for a few days or weeks until I update the documentation (how to use).

CasualPConc update

2009-03-29T18:59:00.000+09:00

I'm almost certain no one has tried it yet, but I spent a little more time to add some more features to CasualPConc, a new parallel concordancer. This application is available at the CasualConc main site under Utility Programs, but the documentation is not up-to-date.

I don't think I'm going make this as fancy as CasualConc, but I'm trying to use much more RubyCocoa (or Cocoa) features (I'm learning...).

CasualPConc originally had just kwic and word frequency count features. Now it has word cluster and collocation features just as CasualConc. One specific feature to CasualPConc is finding keyword in the matched corpus after running kwic search. When you run kwic search, you have the matched portion of the matched corpus (paragraphs or sentences), which includes words that are equivalent or similar to the one you searched. CasualPConc goes through the matched portion of text and calculate keyness of words against the entire corpus. I'm not sure if I explain this clearly, but it's there, but I'm also not sure if this works as intended.

I also added stop word/skip character functions. My understanding is stop words are the ones that are very frequent in a language and eliminating them helps people see what they look for more clearly. You can create stop word lists for any number of languages or corpora. The skip characters function is for two-byte languages, like East Asian languges or more specifically Japanese because characters for period, comma, brackets, etc. in Japanese are not treated as such by regular expressions. They are treated as regular characters like alphabets and included in word lists and context words and they contaminate results. Both of these functions are experimental and not fully tested and they are separate at this moment, but I might combine them as a single function or list.

If you are interested, please try it and give me some feedback. I personally don't use parallel concordancer much and I don't have good parallel corpora, so I can't really test it. Any feedback is welcome (functionality, usability, bug report, etc.). The current version is 0.3, but this simple means I have made two major changes/enhancements with some testing and bug fixing since version 0.1.

Also if you use any of my applications, I'd really appreciate your feedback on them.

A new project

2009-03-26T17:36:00.000+09:00

As I didn't get much feedback and I was kind of busy, I didn't touch any of the programs (scripting) for a while. But I recently did small translation work and thought a parallel concordancer might help in that situation. So I spent the last few days to start a new project. It is a simple parallel concordancer for Mac OS X Leopard and I named it CasualPConc.

Currently it doesn't do much (possibly many bugs) and because I don't really use parallel corpora, I don't have a good idea about how to develop it. So I'd really appreciate any feedback. I used to work as a translator for a short period of time, so if I just follow my intuition, it will be more like a database program for a translator or language learner. The program is available on the CasualConc main site (direct link).

I don't expect many people use it, so if you give me feedback, it is likely that the functions you request will be added (as long as I can handle them). Please email me directly, or leave comment here, or post on the Discussion Board.

A few minor changes

2008-12-28T16:56:00.000+09:00

This would probably be the last update of CasualConc this year.

I made a few minor cosmetic changes and fixed a few minor bugs. The latest version is 0.9.9.2. If you don't have any problem with 0.9.9.1, you don't need to update to this version. I just wanted to make the changes before I forget.

Happy new year!!

Bug fix and few final touch to CasualTranscriber

2008-12-10T10:47:00.000+09:00

I found a bug when I was showing it to someone today. When I added a function to dynamically change menu items based on Preference settings (App Mode), I set some of them to turn off when they should be on. I think I fixed this. I also changed the time stamp insertion. I finally figured out how to override link clicking behavior (links open with a browser by default). So now you can click time stamp on the Editor to go to the time on the movie/sound clip. You can still select the time code and use shortcut (you don't need to use a mouse in this way). The latest version is 0.9.9.1.

But today, my friend showed me there IS software just like CasualTranscriber... It is called InqScribe. Well, what's shocking was it looks almost the same as CasualTranscriber. I mean the layout, functions, etc. I haven't tried it (it's $99 with 30-day free trial, $39 for students), so I don't know how good it is, but it would probably better considering its price and how much time I spent on CasualTranscriber (less than three weeks). So if you are more interested in adding subtitles, you might want to take a look at it. Probably the only advantage of CasualTranscriber is the cost...

Anyway, if you try CasualTranscriber, I'd really like to hear from you. And because there aren't many users (prabably only a few), your feature request might be taken seriously unless it's too complicated for me to handle.

CasualTranscriber Development is wrapping up

2008-12-07T18:36:00.001+09:00

I worked on CasualTranscriber a little more and I finally found a major source of crash. It crashes less now (I hope). I also added a separate player in case someone might want to use controller for your movie playback (such as in class). I also found that OS X can handle MS Word format (only text), so I added a function to handle MS Word document.

I personally think this is a program to help transcription and adding subtitles is a secondary feature, but I looks like my friend uses this primarily for adding subtitles. So I spent a little more time to add subtitles in a different format (not as a text track).

I hope this program is useful for more people. Intended users are language teachers and conversation analysts.

Bug fixes to CasualTranscriber

2008-12-05T16:41:00.000+09:00

Thanks to my friend who is testing CasualTranscriber, I was able to fix many bugs. The program is still not totally stable, but it sounds like it is usable. I implemented almost all the features I can think of (or I can manage) now and from now on I'll fix bugs and might add some features if I get feedback. The current version is 0.6, but it just means I've made some significant changes 6 times since I started working on this.

In any case, if you ever use it and find any bugs or think of something very cool, please let me know. I'm not sure what I do next, but I don't get much feedback, so I don't know what I should do with CasualConc. I want to make it handle 2-byte languages better and possible add parallel concordancing, but it takes time to think about how to implement them. I have a few more ideas about small programs, but maybe I have to get more serious about my own work...

CasualTranscriber

2008-12-01T17:05:00.000+09:00

I know there are not so many people (probably only few mostly I personally know) have tried CasualTranscriber. It's still very early in development, but I've basically rewritten the program from a window-based to a document-based application. Now it deals a rich text or plain text file just like a text editor or word processor. When you close a window, that means you close the document on it.

In addition to the original features (shortcut control of movie/sound clip and extraction of a selected part), you can add chapters, extract a frame image, add subtitles, use the application just as a player, etc. I can't list all the features, but they are all on the new CasualConc site under Utility Programs. Some of the features (esp. adding subtitles) are still experimental and the application itself is not as stable as I want it to be (so now it has an autosave function). This is partly because I'm still new to QTKit (QuickTime Took Kit) and not all QuickTime functions are not avaiable via QTKit. I'm using RubyCocoa, which is a bridge between Ruby, a scripting language, and Cocoa, Mac's application foramework, so there is another layer of issues there.

I want to make this program easy to use for teachers and researchers (to-be) who need to transcribe conversations, speech, movie clips, etc. for teaching/researching. And if you ever use this, I'd really appreciate your feedback.

New Utility

2008-11-20T21:02:00.000+09:00

I started another experiment and started to write a new utility program. It's named CasualTranscriber. It's a simply utility program to assist transcription of movie/sound files (text). I googled and found some good free ones, but because I wanted to learn a bit about QTKit (QuickTime Kit) with RubyCocoa, I wrote it. It's very simple at the moment and at very early stage of development. But if you are interested, please check it out and let me know what you think. If enough people are interested, I will develop it further. Here's the direct link.

As I wrote in the previous post, I started a new CasualConc site on Google Sites. And I want to introduce it here, though only English site is available at this moment (no Japanese site yet). All the program files are still on the old site, but the new site has much more information especially about utility programs (with screenshots). You can access the new site from this link. The CasualTranscriber page has much more information. So please check it out there.

As always, I'd appreciate any feedback on any of the program and also the sites. If you have any suggestion/comment/bug report, please leave comment on this blog, add new topic to the Discussion Board, or send email to casualconc (at) gmail.com.

Bug fixes

2008-11-12T18:01:00.000+09:00

As you might have noticed, the current CasualConc site is hosted on Google Page Creator. But Google decided to stop this service and focus on Google Sites. So I've been transferring site contents to a new site on Google Sites. In the process, I've been updating the content and adding a little more information to some pages. So far, I've created the English site, but haven't started the Japanese site. I'm not sure how many people prefer to have Japanese page, but I will eventually create a Japanese site (I personally prefer to reading in Japanese).

While I've been updating the content, I used basic functions of all the current programs and found many bugs. Most of them were minor and some of them are major, but not the main features. I fixed so many things in the last couple of weeks (as well as adding new functions), so I can't track all the fixes/changes, but here's a list of few of them.

CasualConc:
- Context words function in Concord was broken, but not it should be working.
- Keyword grouping function was fixed
- Keyword grouping only worked when the search was for a group of words. Now keyword groups can be used in a phrase search
- Lemma in search word should work with wild card/phrase search now
- You can import a word list not created by CasualConc now. It accepts CSV or Tab-delimited file with words in one column and frequency in the other. This allows you to import a word list created by other program/script.

CasualTextractor:
- In PDF/Web/Document, most of changes should be undoable. I changed the function to draw text in text area.
- Batch process was not working, but it should now.

I also made minor changes to CaualTagger and CasualMecab, but I forgot what I did. Most of them are bug fixes.

If you downloaded any of the program and find bugs, they might have been fixed now. If not, please report them to me. You can add your comment to the post on this blog or email me or post on Discussion Board (Google Groups). If you have any good ideas/suggestions for any of the programs, I'd appreciate your feedback.

New experiment (UPDATE)

2008-11-11T09:56:00.000+09:00

This is an update to the post, New experiment. I implemented a function to automatically update CasualConcData.ccdb file. So I dropped the beta beta version. The official beta version is 0.9.9.1. Please check the previous post for the details of the last update.

If you try this new function, please let me know what you think.

Link Grammar

2008-11-09T18:13:00.000+09:00

I found this syntactic parser (site) the other day and I also found there is a Ruby binding called Ruby LinkParser (site). The instruction on the Ruby LinkParser looked pretty simple and I thought I would be able to install it and start using Link Grammar from Ruby in no time.

Well, I was wrong. Maybe because the instruction was written for Unix/Linux users, it didn't work on Mac OS X Leopard. I spend half of the day to somehow manage to install it on my Mac. But the process is ugly. I don't recommend it to someone who is not so familiar with Mac OS X system. I hope the original authors could fix the problem and/or write a patch for the latest version of Link Grammar.

Anyway, I put the step-by-step instruction with a lot of screen captures on the CasualConc site. If you are interested, please check this page. If you know or figure out a better way to install them, please, please let me know. In fact, I'm not sure if it's working as it should.

In the future, I want to add a function to CasualTagger to do some simple syntactic parsing using Link Grammar/Ruby LinkParser. But first I need to figure out which functions to add and how. Yoichiro Hasebe wrote a program (port of phpSyntaxTree), RSyntaxTree, that draws a tree diagram from a syntactically parsed sentence (check this page), such as [S [NP RSyntaxTree][VP [V generates][NP multilingual syntax trees]]]. So I want to add a function to output this format (I'm not even sure if Ruby LinkParser does it or not).

By the way, for those who are not sure what syntactic parser is, here's the sample.

The origial sentence is:

Ruby is a dynamic, open source programming language.

If you try to install this, please let me know if it works for you.

New experiment

2008-11-06T17:49:00.000+09:00

I added a new experimental feature to CasualConc. It is an XML information tag handling, which is very limited. What I mean is if you have XML files or XML-formatted plain text files that have information as tag attributes or elements, you can filter the files with the information.

The current implementation can handle two types:

<header attr1="~" attr2="~"></header>

or

<header><attr1>~</attr1><attr2>~</attr2></header>

So if your files have:

<info date="11052008" title="Presidential Election">

or

<info>
<date>11052008</date>
<title>Presidential Election</title>
</info>

CasualConc can preselect the files based on your query. Check this page for more information.

Because of the other changes I made to SQLite database handling, CasualConc is incompatible with the corpora/databases you created in the Advanced File Handling Mode (it shuts down when it tries to read CasualConcData.ccdb file in the ~/Library/Application Support/CasualConc. If you have used a prior version, you need to delete/move/rename it. So this beta of beta is not linked to any pages on the site. If you are interested in testing this highly experimental beta-beta version, please go to this download page directly. (I implemented a function to automatically update the CasualConcData.ccdb file. Now this version is downloadable from the regular download page.) I would really appreciate if you could give me feedback especially on this new function. If you have any suggestion, please make it detailed. I personally haven't used XML files, so I'm not sure if this is useful. More detailed is your suggestion, more likely it will be added to CasualConc (no guarantee, though).

Apart from this beta-beta, I fixed one bug on collocation coloring in CasualConc regular beta version (sounds wierd...).

CasualConc Update

2008-10-22T17:17:00.000+09:00

I've been working on CasualConc and uploaded 0.9.9 to the site. Well, I'm not sure when to put 1.0, so I might go with 0.9.9.1...

The most of the changes I implemented are internal and many bug fixes. I didn't really touch the core tools, so most of the work was done on file handling.

Here are some of the changes you might (or not) notice:

Plain Text File encoding
You can now set a default text encoding on the File view (no need to open Preferences). You can specify a default encoding before you open files, which applies to all the files you add to the file list table. But now you can change them on the table. This means you can select different text encoding for each file. I also added ISO Latin 1 and ISO Latin 2 to the encoding list.

Open/Save panel
This change probably would not make difference to most of the people. I just wanted to change it to Genie panel(?) because I learned how to.

Collocation
Most frequent position for each context word is now colored in red.

Bug fixes
- Exporting/Saving results should work fine now, though I'm not sure how many people have ever tried to use this function. I also made changes to accommodate the changes in Collocation.
- Fixed crashes when you hit space (or may be with some other keys) on the blank table. You might have never done such a stupid thing, but I found this when I accidentally hit the space bar in Advanced Corpus File Handling mode.

Also I put a note on the site, but your preferences settings will be lost if you have used the previous versions. If you want to use it change the name of the preference file "com.apple.rubycocoa.CasualConcApp.plist" in your home -> Library -> Prefereces folder to "CasualConcApp.plist". Except for tag ignore settings, your preferences settings should be safe.

Along with this change, I also added ISO Latin 1 and ISO Latin 2 to the list of encodings (open/save) in CasualTextractor.

If you find any of these attractive or bothered by bugs, please try the latest version and let me know what you think about it. But reports are also welcome.

By the way, I haven't updated all the documentation yet. Some of them are quite old. I guess I have to find time and update them (or rearrange them). I read somewhere that Google is moving the content of Google Page Creatot to Google Sites. That might be a good time to update documentation.

Utility Programs updates

2008-10-18T17:14:00.000+09:00

I've been experimenting some Cocoa UIs and bindings, and adding features I learned to utility programs. I made minor changes to CasualTextractor and CasualMecab. I also made a little more changes and added some new functions to CasualTagger. Now I need to manually tag a lot of texts, so I'm trying to make it a tool to help manual tagging. I copied regular expression replace from CasualConc (hidden feature) and added simple tag coloring. Now it also has a simple word/tag count and kwic concordance of a single file.

I'm also working on CasualConc. As I wrote in the last post, I will probably clean up some old codes. Tag handling might take some time to implement because I have to think about how to handle tags in CasualConc. Any idea? What I'll probably do first is change/fix file handling elements. In utility programs, you can now change the character encoding of plain text files after you add them to the table. This will allow you to use text files with different encodings.

Another minor change is coloring in Collocation. Now the most frequent position for each context word will be colored in red. This looks working fine, so you will see this feature in the next update.

In addition to RubyCocoa programs, I wrote a simple javascript-based parallel concordancer, which I was asked to write. I based it on my old javascript-based concordancer, so not much scripting was involved. I did this because I'm thinking about writing a parallel concordancer for Mac, as I wrote on this blog before, so I wanted to know what are the most fundamental features for a parallel concordancer. I googled and based on some parallel concordancers out there, I wrote it. It only creates a table with matched texts based on the search. It also creates kwic results and you can select one to show the matching text. But what else is necessary for a parallel concordancer for Mac? If you have any suggestion, please leave your comment here or send me email or post on CasualConc Google Discussion Board. If you could give me enough information for me to figure out how to implement your requests, you will have a better chance to see them, though it also depends on my scripting skills.

Anyway, please check out the utility programs and let me hear what you think.

A few bug fixes

2008-10-13T17:49:00.000+09:00

I made a few more bug fixes and some internal changes. The somewhat major but I fixed was aligning and coloring of text on Concord table and text view. If you use CasualConc only with English, you probably didn't see any problem, but if you deal with text with a lot of non-standard alphabet characters, the display was ugly. Now it's better (not perfect). There still is a problem with language that combine more than one characters to display one character on display. Other than that displaying text is less problematic.

The major internal change I made is using Shared User Defaults Controller to save Preference settings. This saved a lot of codes, but at the same time, this is not perfect. Somehow this doesn't remember that changes made by scripts, so for some text data, I have to use script to save the data properly. But I might have done something wrong, so if you find any bugs related to Preferences, please let me know.

I also made a major change to CasualTagger. CasualConc had hidden functions to help manual tagging, which I have never turned on officially. This was because I didn't have time to finalize/fix bugs, so I decided not to make it available. Now I took some of the features from it and added to CasualTagger. I haven't documented them, but I included a simple instruction in the application as a help file. If you are interested, please take a look at it. CasualTagger is on the main site under Utility Programs.

I also made more changed to IPATypist, which not a lot of people use. And I guess those people who have downloaded it might not read this, so they don't know if it's updated or not (though I'm not sure if they keep using it).

That's about it for now. I'm also thinking about adding tag handling features to CasualConc, but it doesn't look promising. I once wrote experimental scripts to handle some types of tags, but they don't work very well. Now if I want to seriously add this feature, I have to get rid of old ones first. It's not very easy... Also there are a lot of weird scripts in CasualConc because it includes some codes I wrote when I was just starting to learn Ruby. I guess I have to clean up old messes first before I add some significant new features.

In any case, if you use CasualConc and/or other utility programs, please let me know what you think. The current priority is adding tag handling features. East Asian language support might be dropped. It would be a separate program. Some people asked about parallel concordancer, so I might write a separate one for it, but I still don't have enough information to go ahead. If you'd like to see a parallel concordancer for Mac, please give me information. You can email me directly or make a comment on this blog or post on CasualConc Discussion Board. I need to know what are the most fundamental features and how they should be implemented.

IPATypist update

2008-10-10T17:52:00.000+09:00

This is not related to corpus analysis, so I guess almost no one is interested, but as my memo, I write down what I did.

IPATypist is a very simple utility program to type IPA phonetic alphabets. With this update, I added a database function to it. Now phonetic transcriptions can be stored so you don't have to type them again. But this is mainly my experiment to use CoreData, an OS X framework to easily handle database type programs. I just started look at it, so I'm still not sure if I did this right or wrong, but it looks it's working.

If you happen to be one of rare people who are interested in this program, please check it out and let me know what you think.

Utility Programs

2008-10-09T16:08:00.000+09:00

I updated a couple of utility programs, including changing names, and added a new utility program to the main site.

Two of the utilities are TextExtractor and jparser. These two names are already used for program/module names, so I decided to change them. I use the same Casual~ namings for them and TextExtractor is now CasualTextractor and jparser is now CasualMecab. Well, it's obvious I didn't spend much time on this... Both utilities had a few bug fixes and minor feature changes, which are not obvious.

And a new addition is a POS tagger. I finally found a English POS tagging module for Ruby. The module is EngTagger by Yoichiro Hasebe, which is a Ruby port of Perl's Lingua::En::Tagger module. I simply added GUI and a function to process multiple files. You can also select a tag type (the default of EngTagger is xml format). I named it CasualTagger and you can download it from the CasualConc main site (here's the direct link to the page: English/Japanese). To use CasualTagger, you need to install EngTagger, but it's very easy from Terminal.app (single line of command). For more information about EngTagger (tag sets, etc.), check this page. As always, any feedback is welcome.

Now I need to seriously think about adding tag-handling feature to CasualConc. But how?? If you have any suggestion, leave your comment on this blog or email me (email address is on the main site).

Also a couple of people asked for a parallel concordancer (for Mac and with Javascript). But what's the most fundamental functions? Any suggestion/comment about parallel concordancer is also welcome.

TextExtractor

2008-09-19T09:24:00.001+09:00

I haven't had time to write any script at all for a while, but I've found some time to experiment on RubyCocoa. The stuff I'm working on won't change anything on the surface of CasualConc. The changes will be mostly internal and slow.

A very few feature requests I got are xml file handling and parallel concordancing. But I don't have any experience in these, so I need more information. I added two threads to Google Groups discussion board about these two. If people ever read this and give me information about these two, I might add them or write a separate program for them (parallel concordancer).

Another thing I'm considering about CasualConc is dropping East Asian Languages support. I don't hear from anybody who uses CasualConc for Japanese/Korean/Chinese, so I don't know if I really want to keep trying to accommodate these function on CasualConc. It would probably be easier for me to maintain if I separate East Asian Languages concordancer (eps. kwic) as another program. I'll think about this more, but if you have any suggestion, let me know.

Anyway, as a part of my experiment on RubyCocoa, I updated TextExtractor, an utility program to extract text data from verious text embedded files and to convert text encoding of plain text files to UTF-8. I'm not sure if you looked at the utility program section of the CasualConc site, I have a few utility programs that deal with text files. I combined two of them (PDF to Text and HTML to Text converters) and added a few extra functions. The first version (0.1) of TextExtractor had a function of jparser (a Japanese parsing program using MeCab), but it didn't run without MeCab and MeCab-Ruby. So I dropped this function.

Instead, I made that part to simply convert non-UTF-8 text files (.txt) to UTF-8 text files and MS Word, PDF, HTML, OpenOffice documents to UTF-8 text files or Rich Text Format files. All other parts (PDF to Text, Web file to Text, and batch process) can save files as RTF files. When you convert files to RTF files, you can either keep text/font information of the original files (fonts/font style/etc.) or throw away this info and save as a plain text on RTF file.

I also added basic instruction in English (not translated to Japanese yet). So if you are interested, please try it and let me know what you think.

EDIT: This program is renamed as CasualTextractor

A few very minor changes

2008-09-07T14:25:00.000+09:00

Well, obviously, I haven't done anything with CasualConc for the last three months. I finally announced this on Corpora List and some people got interested and started testing CasualConc. But I heard from only a few people. Still it's good to know someone uses it and likes it.

I made a few minor changes/bug fixes to CasualConc. The only one that's worth mentioning here is that now CasualConc remembers the files you selected in File Mode when you quit the program. The next time you start CasualConc, the files you selected last time should be on the file list. Now the version number is 0.9.8 beta.

As always, I'd like to know what you think about the program.

Bug fixes

2008-06-08T01:32:00.000+09:00

I finally got a bug report. Now I know at least a few more people are using CasualConc.

The bugs are related to the recent changes I made to Lemmatization and Collocation.

The bug related to lemmatization was that when lemmatization was activated without specifying a lemma file, CasualConc crashed. This was because CasualConc looked for a lemma file when it started or returned from the preferences and if the file was not found, it crashed.

The two bugs related to collocation were 1) it didn't run in file mode, and 2) search in concord didn't work when 'Treat Keywords as One Word' option is activated in preferences. These should be fixed and work properly now.

I would appreciate any report of bugs. And I'd like to know how you like CasualConc.

A minor change

2008-05-30T10:31:00.000+09:00

I found a bug (sort of) in Concord a couple of days ago. It's a minor bug and this happens only when you use the database mode in Concord. Well, it's more of memory leak. I implemented a forced garbage collection when full text is displayed in the context view of Concord, but somehow memory is not released. So I changed the way to read the text from a database file. Now it should not keep using additional memory when you select a different concordance line to show full text.

I use the same technique to read data from a database file when CasualConc searches a string, but if I implemented the same change to the search function, it used more memory because the search returns more hits. What this means is if you search word(s)/phrase(s) in any of the tools many times, CasualConc keeps using memory. I haven't tested if it uses up all the available memory and starts using virtual memory or if Ruby starts GC when it uses up all the available physical memory. In any case, until I can find a way to solve this problem, you might want to quit CasualConc after a while and restart it.

A few more fixes again

2008-05-20T11:46:00.000+09:00

I fixed a few more bugs this weekend that are mainly related to lemmatization and collocation statistics. I also added some more documentations to the main site (some in Japanese). The latest version is still 0.9.7 but the date is 05192008.

Now, most of the features I wanted to include in CasualConc is there and mostly functioning. I don't have time to improve Japanese kwic feature now, so that should wait until sometime in summer or fall. And unless I find or someone reports any major bugs, I will try not to spend too much time on this for a while. I don't know how many people actually downloaded CasualConc and are using it, but I guess there aren't many. If you happened to be one of them, I'd like to hear what you think about it.

Well, I might need to publicize this a bit more, so I might start trying to get more beta testers somewhere.

Another bug fix and minor update

2008-05-18T06:33:00.000+09:00

I highly doubt if anyone downloaded CasualConc recently, but anyway, I found a few bugs and also make some changes, which I wanted to for a while. Now the latest version is 0.9.7 beta.

First, I found a bug in Japanese Concordance, which I'm sure nobody has ever used. When I dropped the text only mode, I forgot to change it in Japanese concordance mode. Now it should be fixed. I also fixed some other bugs what relate to the recent feature changes.

The changes I made are mainly with Collocation. Now, if you search for multiple words or use wildcard search and multiple words are found, collocation information will be displayed for each keyword. This change affected statistics calculation, so I think I made necessary changes to it.

I also made a minor change to Export result function of Concord. Originally, an exported CSV file from Concord only include kwic results and file paths. Now it has an option to include context words (L5 - R5). To include them, go to Preferences -> Concord and check the box Include context words (L5 - R5) in CSV output.

As always, if you happened to find this blog or the main page, and tried CasualConc, any feedback (including bug reports) is welcome. Especially if you find it useful, I'd like to know.

Another quick fix

2008-05-15T11:55:00.000+09:00

I found a minor (maybe major to someone if anyone ever uses CasualConc) bug and fixed it today. This only affects you if you use Concord with non-plain text files as your corpus files. And this only happens in the paragraph mode (the default mode). Now you should be able to use other file types as your corpus files with Concord and in the paragraph mode.

I found this bug when I was testing .odt files. After the fix, I was able to use .odt files as corpus files, so this confirms CasualConc can read .odt files!

I don't know how many people are affected (I know not many people), but if you downloaded CasualConc in the last couple of weeks, please go to the site and download the latest version. It has the same version number (0.9.6), but different date (05142008).

And if you find any other bugs, please let me know. The email address is on the main site.

Some details of last update

2008-05-13T05:15:00.000+09:00

As I mentioned in the last post, I added/activated a couple of new features on CasualConc. One is based on the lemmatizing function and the other is something with Concord.

The first, which is based on the lemmatizing function is keyword grouping or whatever name I will settle (it has a tentative label). What it does is first you prepare a text file (UTF-8) with the same format the lemmatizer accepts. The default is:

keyword -> word,word,word,...

The keyword is a grouping label, so if you want to group days of a week, it looks like:

week -> Monday,Tuesday,Wednesday,Thursday,Friday,Saturday,Sunday

Once you prepare as many groups you want to have, save the file as a plain text with UTF-8 encoding. Then, go to Preferences on CasualConc -> Lemma, and check Grouped Keywords. Next you select the file you just saved by clicking Select Grouping File button. Now everything is set.

If this works as intended, you should be able to use this function on Concord, Cluster, and Collocation/Cooccurrence. What you will do is add @@ at the beginning of your search word(s). So if you want to search all the days of a week, as specified above, you will @@week, then search. You should be able to search all the words in this group. Technically, you should be able to search multiple groups, but it is not fully tested and might not work, and I don't know what will happen if you combine this feature and wildcard search. I might change the behavior of this feature if I ever get any feedback.

Another somewhat major addition is which is not documented at this time is a function for Concord. You can now open a concordance result in a new window. This might be useful if you want to compare several concordance results. To use this function, search any word(s) in Concord, and then go to Menu -> Misc -> Open Concordance Result in New Window. This is experimental. I added this because I found a way to add multiple window function to a program (I just wanted to have something so that I remember how to do it). You should be able to resort the results even on a new window, just like on the main window. But be ware, if the concordance result is huge (like returned 10000 hits), using this might eats a lot of memory because CasualConc keeps all the info on memory. If you have at least 2GB of memory, this should be less of a problem, though.

Finally, I have something that is not related to CasualConc. I posted a couple of weeks ago that I wrote a simple utility program that helps typing IPA characters. I wrote a similar program(?) with Javascript and added to the IPATypist page. I highly doubt many people read this post and especially people who don't use Leopard, but this is written for those people. It should run on Tiger with Firefox, Safari and Camino. I haven't tested it on IE on Windows and I have no intention to support it, but it might work. It is also available for download, so if you are ever interested, you can download it and use it on your computer or put it on your course site or wherever you want to use it, though I can't guarantee it will work.

As always, if you ever use any of the programs, I'd apprecite your feedback. That will motivate me to improve them.

Minor update

2008-05-06T16:35:00.000+09:00

Over the weekend, I fixed a few bugs and made a few changes to some of the existing functions of CasualConc. But these changes might have introduced another bugs... Now CasualConc is 0.9.6 (beta).

The bugs or more precisely, legacy features, that were fixed or updated were mostly on file handling. When I first started writing the program, I didn't know anything about RubyCocoa (or Cocoa). So when corpus files/database file were selected, only one file/folder was selectable. This was simply because of the original Ruby script. In that script, I simply specify a directory or a file to analyze in the script. And I simply added Cocoa interface to it. Eventually, I learned how to receive multiple file names as an array from the open panel, I made it available to some of the new features. Now you should be able to select multiple files/folders when you choose your corpus files/folders in File Mode. In Database Mode, only one file can be selected. If you need to select multiple database files, please use the advanced file handling mode.

Another bug fix was drag and drop of files. I'm not sure if I mentioned this feature in any of the documentation, but you can actually drag and drop files to the table in File View. So if you have a Finder window open with files you want to add, you can drag and drop them (or the folder that contains the files) to the table. This should work with files to analyze in File Mode and files to add to a database in Database Mode. If these don't work, please let me know.

Other minor changes are too minor and I'm almost certain no one has ever used it. But anyway, I dropped Text Only mode. So now all you need to do is check/uncheck the file types you want to use. You still need to specify text encoding if you use text files. This is because the auto-detection of text encoding in Objective-C is not usable. Related to this change is the addition of OpenDocuemnt Text (.odt) support. But because I've never used Open Office, I haven't tested it. I implemented this a while ago when I added others but didn't activate it because I don't use it. And now I decided to activate it. I simply use a bulit-in Objective-C function, it should work as other files do (no guarantee).

Oh, one, kind of, major fix is the lemma function. I implemented the lemmatization function at a very early stage. But I've made a lot of changes to most of the tools since then, so it seemed like I broke it. Now it's fixed and I also added a function to use lemma grouping in kwic search. I mean, if you turn on this feature and search a word that is on the lemma file you provide, you can search all the words grouped under the same lemma, though I'm not sure if this works as intended.

In addition to these mostly fixes, I added a couple of new features. One is based on the lemmatization function and the other is something very experimental. But this post is getting long, so I'll post them in the next few days when I have time.

As always, if you happen to find this blog or the CasualConc site, please let me know what you think. You can leave your comment on this blog or email me (the address is on the main site).

IPATypist

2008-05-03T16:14:00.000+09:00

This is a small utility program I wrote for an ESL instructor at my school (yes, this is just written for you, Janet!), but I made a few changes to it so that this can be also useful for other people.

Originally, she told me she was having trouble typing IPA phonetic alphabets in Unicode. There is a keyboard mapping to type phonetic alphabets, but it is cumbersome. So I simply put a lot of buttons to enter IPA characters. Because this was written specifically to serve her purpose, the characters are the ones used for English and some special ones that are used for the book she and her colleagues are working on.

What you can do with this utility is type phonetic alphabets by simply clicking buttons. Once you are done, copy/paste them to whatever the program you are working on. You can either go to Menu to copy or Command + C to copy the string, which keeps all the font information (font type/size). Or you can click Copy button, which only keeps the character information, so when you paste the string, whatever the font setting (type/size) on your document will be applied.

The latest version (0.3) supports key (button) mapping (if it functions as intended). Now, any character can be assigned to any of the buttons, so users/teachers of languages other than English could use it (I hope).

The system requirements are the same as all the program/utilities I wrote: Mac OS X 10.5.2 (Leopard) or later. I think this works on 10.5, but now all my machines are running 10.5.2, so I can't check the version prior to this (but at least I'm sure this won't run on Tiger). You also need Doulos SIL font, which can be downloaded freely from the SIL site. The link to their site is on the download page of this utility.

If you find any bugs or have any feature request, I will try to fix them/add them as much as I can (if they are minor). I don't have time to spend much time on this now (or I should say I should not spend time on this). But any feedback is welcome. Especially, I'd like to know if this helps someone.

Documentation

2008-04-29T11:13:00.001+09:00

Over the last couple of weeks, I've worked on the documentation of CasualConc. Now, it covers most of the basic functions. I also started more step-by-step instruction with a lot of images and named it Getting Started with CasualConc. So far, I have only finished basic file management and database creation along with the kwic concordance function, which I think will be the most frequently used function (I do).

Now, my only hope is someone will find the site or this blog and start using it. Somehow, I can't search the CasualConc main site on Google. It doesn't show up in the result. When I add a post to this blog, it shows up in the next 20 hours or so and disappears. Well, maybe I should add one post per day until some more people find this blog and CasualConc.

If you happen to find this blog, please try it (if you use Leopard) or tell your friends who uses Leopard to try it. I know it still has bugs and a lot of limitations, but I really want other people's opinions to improve it (it serves most of my current uses, so I don't have much motivation to make a lot of changes). Well, even if I hear from people, I might not be able work on it for a while, but at least it's good to hear esp. if people like it.

A quick fix

2008-04-28T02:55:00.000+09:00

I'm almost certain nobody has downloaded CasualConc since my last post. But anyway, accidentally, I introduced a bug to database creation function. This was caused by implementing a new tag-deletion code, which I forgot to apply to database creation part. So if you find the database creation function does not work properly (this crashes CasualConc), please go to the site and download the latest beta.

If you find any other bugs, please report it to me. The email address is on the main site. Or you can leave a comment on this blog.

Thanks!!

A few more fixes

2008-04-27T01:57:00.000+09:00

I found a few minor bugs which I introduced with the last changes, so fixed them. And I also found that the default font of CasualConc, Courier, is not monospace in Greek, which is the language my very first user tests (I guess) on CasualConc, so I added a function to select Courier or Courier New, which is monospace in Greek.

I didn't mention this in the last post, but I also made some changes to the codes of Concord, which only improved speed about 2-3%.

Now I hope more people find this blog or the main site and test CasualConc. So if you happen to find this blog or the main site and you know someone who uses Leopard and is interested in corpus analysis, please tell him/her to test CasualConc, even if you don't use Mac OS X Leopard. If you do, please try it!!

A minor update

2008-04-26T02:03:00.000+09:00

As I reported a couple of days ago, at least a few people around the world are testing CasualConc and I've already got a report of a bug... This very minor update is partly based on the report, which I don't really know the source of, and a minor change to a setting.

What I found was an inconsistency of handling special characters, such as curly quotes or curly apostrophes. These look like a single byte character on web pages or Word documents, but in fact two-byte characters in Unicode (UTF-8), so CasualConc replace them with a single byte quote or apostrophe. Recently I added .doc and .pdf support, and these documents often contain their own special characters (arrows, etc.). I only replaced these in some parts and not others, which caused inconsistency. Now I think I applied the same rule to all the tools, but I'm not sure.

Another change is drop of ASCII mode in concordance. In the Concordance Preferences, CasualConc has 4 ways to handle texts. Originally the two European Language supports are ASCII and with Acccented Characters. The former assumed the corpus files do not contain any multi-byte characters (in UTF-8). The latter only assumed a few accented (multi-byte) characters in a context (the context words shown in the concordance result table). But then I realized after the very first person downloaded CasualConc that he uses Greek, which, I think, uses full of multi-byte characters in UTF-8. So the new two modes for European Languages are A and B. A is the same as the previous with Accented Characters mode and B is for full multi-byte character languages, but still assuming not many 3-byte characters used in East Asian Languages. If the text contains many 3-byte characters like East Asian Language characters, like Japanese, which are 2-byte characters on the screen but processed as 3-byte characters in UTF-8, CasualConc might not be able to display concordance result or full context view properly. If there are any languages that have full of 2- and 3-byte characters, let me know. I'll see what I can do.

By the way, I decided to add 'Getting Started' section on the CasualConc site. The current 'How to use' is more like a manual or lists of functions CasualConc has, so it's not really how-to. The site only has basic file handling or 'how to select files for your analysis' type entry. I'll try to add more when I can find time.

Anyway, the current version is 0.9.4. If it is up to 0.9.9 and still not ready for version 1, I might go for 0.9.9.1..., but if enough people use it and does not have major problem, I might put it as Version 1.

Old stuff

2008-04-24T23:04:00.000+09:00

Now I learned I can add an html page with javascript to the Google Page Creator site by simply uploading it as a file and link to it, I added an old javascript-based concordancer/word counter to the CasualConc site. This is probably useless for people and I'm not sure if I need it on the site, but I just wanted to keep it somewhere and because this old script is the basis of CasualConc, I think it's the right place (for me).

I wrote this script about 2 years ago when I was playing with javascript. At that time, I wanted to learn javascript, which I just started to learn a few months before that. I only knew MS-BASIC before that. When I started to play with javascript, I figured the best way to learn it is to write something with it. I first wrote s few scripts for my colleague at the work to save a repetitive task. Then, I wanted to do something for my self. I always wanted to do something with corpus linguistics. I found a few sites that did it with javascript and many scripts in Perl. With trial and error, and a lot of revisions, this javascript page was written. The page says it's version 6, but the script version is 61 (its on the file name of the script file). But then I learned limitations of javascript as a tool for corpus analysis. Then I tried Perl because that seemed to be what everyone used (and a lot of people are using it for text analysis), but somehow, it didn't appeal to me (or I wasn't/isn't smart enough to learn it). Then a year later, I used Ruby for something at work and somehow I liked it (still like it). I didn't know Python, which I learned when I was learning Ruby. Another big plus was that because Ruby was originally and is still developed mainly by Japanese people, I found a lot of documents in Japanese. This and the inclusion of RubyCocoa in Leopard is why CasualConc exists now. I think I wrote something like this in the very first post on this blog, but anyway, it's fun to use Ruby though my scripts are still primitive. I hope I can learn more about Ruby and improve CasualConc. What I want is time, but now I need to spend more time on other more important stuff...

CasualConc launched!!

2008-04-23T16:17:00.001+09:00

I finally found someone who got interested in using CasualConc!!. I was just surfing the web looking for info on concordancing on Mac. Though I'm developing CasualConc, if I can find better more flexible concordancer for Mac, I'm happy to use it. The only problem will be I can't make the changes I want. Anyway, I found a blog that was describing poor concordance software situation on Mac, so I posted a comment and he replied to it and wrote he downloaded CasualConc. He wrote he would post the impression of CasualConc on his blog, so I'm really looking forward to it and at the same time I'm a bit nervous. I think CasualConc at its beta state works ok for my casual use right now (mostly searching for collocation of words I want to use in my paper). With database mode, it's fast enough to use regularly. And because I wrote the program, I know how to use it, but I'm wondering how easy or difficult CasualConc is for others. I've been adding contents to the documentation. But at some point, I might need to work on step-by-step instruction of how to use it. Well, this will only happen if more people are interested and start using CasualConc.

If you ever find CasualConc and use it, any comments are welcome!

Japanese Support

2008-04-21T09:03:00.000+09:00

This would probably the last update for a while. While I was stuck with the idea for my dissertation, I simply spend some time here and there for the last few days to make minor fixes and feature enhancements to CasualConc. As I posted in the last couple of days, I finally made the download page available to public, though I'm not sure how many will find it, and added support for several different character encodings and file types. Finally, I added very limited Japanese support.

Now CasualConc can read Japanese (and possibly other East Asian Languages) files in two formats. One is a plain format without any space in between words. The other is wakachi-gaki, which has 1-byte space in between words. Wakachi-gaki files can be created with jparser unitility program. To analyze Japanese texts, a proper mode should be selected in the preference. Select Japanese (plain) under Concord options in the preference for the former and Japanese (wakachi) for the latter. If a proper mode is not selected, CasualConc cannot search words/characters. Wildcard search is implemented, but not tested thoroughly. Because of the way wakachi-gaki is written, 1-byte space should be inserted between words in phrase search. Because this is also experimental, CasualConc might crush when you try to analyze Japanese text. Japanese is only available for Text Mode. Once features are set, I will add database file support.

If you happen to find this blog or CasualConc page and are willing to try, please do so and let me hear what you think.

CasualConc open to public

2008-04-20T04:08:00.001+09:00

Well, I finally decided to make CasualConc public. This just means I added a link to the download page, which was already active, to the home page. I highly doubt anyone is visiting the CasualConc site, so this doesn't make much difference, but I'm hoping somehow someone might find the page and try it. When I googled, it didn't come up, so the only way to find the page is from a bbs (or usergroup?) post I wrote while ago (which I mistakingly posted multiple times because I thought I was able to edit my post, but it turned out I posted multiple times...) or from a link on my personal schedule page at my work. Or possibly, from this blog, if this can be googlable.

Anyway, because I don't get any feedback on existing features, I decided to work on something not currently implemented: Japanese (or Asian Languages) support. This is going to be highly experimental and I don't have much time now, so I can't tell when I will release it. So far, I can display kwic results of Japanese text in plain format (no space) and wakachi-gaki format (space-separated). The former can be sorted by L5-R5 context characters and the latter can be sorted by L5-R5 context words (or whatever the separated units are). In the future (only for Japanese), I want to include MeCab (which you need to install following the instruction on the CasualConc page) to process plain texts, but this won't happen near future.

If you ever find this blog and use Mac OS X Leopard and are interested in corpus analysis, check CasualConc and let me know what you think. The link to the CasualConc site is on the right or click this link.

Another file format support update in CasualConc

2008-04-19T03:14:00.000+09:00

After I updated CasualConc last night, I realized I could add html and WebKit Webarchive support. So I added these two to supported file format. Now it can read various files that contain text. But the process will be slower than plain text files. So if speed is important, convert the files to plain text. If you want faster search, then create a database file from plain text files. I will try to write a utility program to convert CasualConc supported files to UTF-8 plain text files.

Well, it's kind of sad to keep writing blogs knowing nobody is reading...

CasualConc update

2008-04-18T18:56:00.000+09:00

With recent discoveries of file handling on Objective-C side (or RubyCocoa side?), I decided to add a new feature to CasualConc. Originally, CasualConc was able to handle UTF-8 or ASCII encoded files because I used Ruby's file handling method. Now I switched to Objective-C methods, so it can handle a few more encodings. Added encodings are UTF-16, Windows Latin 1, Windows Latin 2, Mac Roman, Shift-JIS, EUC-JP, and ISO-2202 JP. The last three encodings are all for Japanese. These are limited to the ones Objective-C can handle by default. I wasn't able to find Chinese or Korean encoding settings, so they are not included. But CasualConc cannot handle 2-byte character properly (in Concordance), so this shouldn't be an issue. I haven't really tested all the encodings, so if someone happens to find this blog and would like to try, let me know. I might add a link to download page to the CasualConc site soon (hopefully).

Also experimentally implemented is support of other file formats. This still returns error from time to time. Personally, all my corpus files are in text, so this is not for myself, but I just thought someone might be interested. The problem is no one is checking this blog or CasualConc site, so I highly doubt anyone even uses this function. By the way the added file formats are .doc, .docx., .rtf, .rtfd, and .pdf.

Well, I really hope someone would try to use CasualConc, though...

WtoTconvUtil

2008-04-13T16:17:00.002+09:00

This is another utility program I wrote in Ruby + RubyCocoa. What you can do with this program is very simple: extract text from a web page. In fact, I'm not sure how useful this is, but I just wanted to experiment. This is based on PtoTconvUtil, a PDF to text converter utility program. After I wrote it, I wondered how I could extract text from a web page, so I experimented a while, but couldn't figure it out. But last night, when I was not able to think about my paper, I found a clue at a web site, and then spent 15-20 min. to figure out how.

This program still has many issues because the browser part is simply made of Cocoa binding, which means no scripting. I simply wrote scripts to extract text and save it as a text file. But thanks to built-in Cocoa functions, it recognizes a web address in a text box (though you always need to type "http://"), reload, forward and back buttons work, and accepts Safari Webarchive file and HTML file by drag&drop. And I also found that this program can extract text from a PDF file which is displayed on a browser with a plug-in. So if you know a web address of a text-embedded PDF file, you can show it on a browser box and extract text. Yes, this is very good, but because I just use built-in functions, it's not flexible. I want to add a function to read bookmark from other browsers, so you don't have to type an address everytime. It might be easier to read Safari Bookmark, so I might try it first, though I'm not sure when that will happen.

Anyway, this program helps you compare the original and the extracted text on one window. So if you build a corpus from web pages, you can either extract the entire text or simply copy and paste a part of it.

So again, if you somehow found this page and is reading this, AND if you use Leopard, try it and let me know what you think.

EDIT: This program is discontinued and integrated into CasualTextractor, which is available on CasualConc site under Utility Programs.

CasualMecab

2008-04-12T17:19:00.001+09:00

is the name I gave to a utility program that is based on MeCab. What this program does is POS/morphological analysis of Japanese text. What the program does at this moment is simply produce MeCab output. Choices are MeCab output, Chasen-like output, wakachi-gaki (words with spaces in between), and yomi (in katakana). The output can be saved as a text file. I want to add other output formats, but probably not in the near future. This program can also handle batch process although I haven't tested it extensively. The output file is encoded in UTF-8, mainly because that's what CasualConc can handle. I want to add Japanese concordancing feature to CasualConc in the future. If anyone ever finds this blog and is interested, please go to CasualConc site and download it. By the way, this program requires MeCab and MeCab-Ruby. The instruction to install these are also at CaualConc site. The installation is not simple (you need to use Terminal and command line to install), but the instruction is step-by-step. I hope anyone can understand it. As always, this is a Leopard only program and free.

MeCab-Ruby

2008-04-11T13:14:00.000+09:00

I finally found a way to successfully install MeCab (Japanese parser) and MeCab-Ruby, Ruby binding for MeCab on Leopard. I added this page to the CasualConc web site. It's only in Japanese at this moment because I'm not sure how many people actually check the site and how many of very limited visitors are interested in installing MeCab-Ruby on their Leopard machine. If anyone is interested, I can translate the page into English, but probably there are many better sites somewhere.

But now that I installed it, I might add Japanese concordancing features to CasualConc, if I ever have time. At least, I can try it now. Also if anyone can understand how to install MeCab-Ruby on their computer, I might add parcing feature (Japanese) to CasualConc, assuming people are willing to install it on their own. But I'll probably first work on GUI interface of MeCab-Ruby to create wakachi-gaki files or syntactically parsed files. But when do I have time???

Garbage Collection again

2008-03-14T06:10:00.000+09:00

After some experiments, some of my efforts paid off, but not all. Then, I realized it was not just Ruby that used memory. Becaue it's written in RubyCocoa, Cocoa or Objective-C part should use some memory. I believe Objective-C 2.0 has GC, but OS X uses as much memory it has and manage it. I might be wrong, but using more memory itself might not be that bad.

Concord, Cluster, Collocation might be usable with modest amount of memory, but Word Count (n-gram) requires a lot of memory. This is because it creates a huge array (all create arrays, though). I know my current implementation is not ideal, but maybe I have to improve Word Count first. When I was testing the original Ruby scripts, I only used smaller corpora (far less than 100 mil.). Now I need to figure out a way to reduce memory usage, but how? Does anyone have good idea? My implementation is to use hash to count, just as any basic Ruby book shows. But I tweaked it a bit to increase processing speed.

Anyway, this is partly why I put CasualConc can handle 1 mil. corpus at reasonable speed. Well, I need time.

Garbage Collection

2008-03-12T11:44:00.000+09:00

I use CasualConc regularly to look up how certain words are used in a context. When I was using it, I realized CasualConc is a memory hog. I knew Word List, espcially when used for n-gram list, needs a lot of memory to process because it keeps counting new ones while it stores counted ones (not exactly, though). I knew Ruby has garbage collection built in, but it seemed like it wasn't working when I wated it to work (maybe because there still was a lot of unused memory). So I decided to force GC to start at some points (GC.start).

But when?

I've been trying several differnt points per each tool and associated method and monitor the differences. But because I've never seriously studied programming (I'm not and have never been in computer science), I don't think I understand how GC works (or in fact, I'm still not sure what exactly OO language entails. If you are breave enough to take a look at the Ruby/RubyCocoa source code of CasualConc, you can see my scripts are not written in Ruby way. I hope I have some time to learn to program a little more seriously someday, but for now, CasualConc works ok (at least for me).

Anyway, I'm not sure if someone ever reads this entry or any entry on this blog, but I'll try to keep my record on this. I want to add some memos on Ruby/RubyCocoa codes on this blog if I can.

PDF to Text converter

2008-03-09T05:12:00.001+09:00

In the last post, I wrote I found a way to extract embedded text from PDF files. I wanted to do something with it before I forget, so I wrote a simple utility program in Ruby+RubyCocoa and posted it to CasualConc site. The system requirement is Mac with Leopard. I named it simply PDFtoTextConverter. What it does is open a PDF and show it's embedded text in the text box on the same window. The extracted text can be saved as .txt file. It also has a batch process mode. You can add PDF files to the list and select a folder to save the text as .txt or save .txt file to the same folder where the origial PDF files are stored. If you are interested, please try it. You can go to CasualConc site by following the link on the right.

EDIT: This program is discontinued and integrated into CasualTextractor which is available on the CasualConc Main site under Utility Programs.

PDF

2008-03-07T02:37:00.001+09:00

I finally found a way to extract text from text-embedded PDF files in RubyCocoa. I personally don't care about this much, but I guess this might be useful for some people. The problem with handling PDF text is not the extracting part. I mean, the real issues with implementing it to CasualConc are:

1. each line of text is separated by a line feed character LF (\n or \r\n?)
2. page headers/footers, etc. that are not the main text are also included
3. embedded text often includes extra spaces, garbled characters (often with ligatures), etc.

1 is probably the main issue. Currently, the basic unit of analysis in CasualConc is paragraph, which means text separated by LF characters. So it cannot handle text files that separate each line with LF characters such as Brown Corpus files. This require some coding (means not just adding a few lines) and I can't find time to do it now. I'll try to implement this feature in the future, but I don't know when.

2 and 3 cannot be avoided, I guess. So I might try to add a feature to extract text from PDF files within CasualConc, but this also requires certain amount of time.

But at least I know how to extract text from PDF files. So the feature will be included in a future version of CasualConc. If many people are interested, I might prioritize this (but probably won't happen at least until Summer).

Google Search

2008-03-06T18:19:00.000+09:00

I've been adding documentation to CasualConc site, although I haven't yet added a download page. Now it has a page for Concordance and Word Cluster along with Basic File Handling.

But now I'm wondering how Google works. I mean the Google Page Creator Help says the page created by it "can be crawled by Google within a few hours of publication". Well, it says "can be", so the actual time might be longer than a few hours. In fact, the CasualConc site was searchable on Google a couple of days after I published it. BUT now it's not on the search result. It disappeared!!

Maybe I should tell my friends to check this first...

CasualConc

2008-03-03T17:31:00.001+09:00

I started this blog to keep track of what I do for CasualConc, experimental concordancing software for Mac OS X 10.5 Leopard (and possibly later version of OS X).

I started to learn a scripting language called Ruby, which is similar to Perl or Python, last summer. The main reason I chose Ruby was that there are many documentations in Japanese. I don't know if I made a right decision, but at least I tried Perl and Ruby but I like Ruby better for no particular reason (Perl simply didn't appeal to me when I tried). Another reason was that I read somewhere that Apple decided to include software (?) that bridges Ruby and Cocoa, Mac OS X's GUI framework (?) in Leopard. It's called RubyCocoa and it allows users to add Mac GUI to Ruby scripts (btw, there's a similar one for Python). Isn't this cool?

At first, I used Ruby for my work (I'm working as Instructional Technology Consultant at my school), but later decided to learn it more seriously. I'm interested in corpus linguistics and want to do some corpus-based/driven research, so I decided to write some scripts for basic corpus analyses. When OS X 10.5 Leopard came out, I had a few simple scripts for kwic, word count, etc., so I tried to add GUI to them. It wasn't very easy because there isn't much documentation for RubyCocoa. So I had to learn both Ruby and Cocoa and combine them to make GUI work.

Now, I have added some more features to kwic and word count and named it CasualConc. It is Mac GUI based software written in Ruby+RubyCocoa. Because the developing environment is OS X 10.5 Leopard, it only runs on Leopard. There might be a way to make it run on Tiger, but I don't want to spend time on it simply because I don't have time (and I don't have expertise). The current version is 0.9 and still in beta (well, beta simply means I call it so). I don't have much time to make a lot of changes now. From now on, I try to fix major bugs and write up some documents. And now I want to have someone to test it.

There is no guarantee that this works for you, but if you are interested, I'm happy to have you as a beta tester. Here's basic info:

System requirement: a Mac with a lot of memory (at least 1GB) and that runs Mac OS X 10.5 Leopard (Universal, well, this is mostly written in Ruby...), optimized for screen at least 1280px wide (13.3 inch or larger on notebook or 17 inch or larger on desktop LCD)
Acceptable file format: text files (.txt) encoded in ASCII or UTF-8 (Ruby is not good at handling character encodings)
Acceptable languages: any single-byte character language (double-byte character languages (East Asian languages) can be analyzed except for kwic concordancing as long as words are separated by single-byte space)
Target User: Mac users who don't want to start up Windows machine, switch to BootCamp, or run Virtual PC/Parallels/VM Ware for simple concordancing for preliminary analysis, preparing teaching materials, learning, etc. (CasualConc is probably not good enough as your primary research tool)

I use CasualConc on my Mac mini (1.86GHz Core 2 Duo) and have used it on G4 (1.5GHz) machine. It works fine for me, but with faster CPU and more memory, performance is better. With 1 million corpus, it works at reasonable speed (not as fast as WordSmith Tools). With a corpus larger than that, well, you can try.

If you are interested, check out CasualConc site. Documentation is not complete (far from it), so if you have never used any concordancer, you might find it difficult to use. But if you have, you can probably use most of the basic features.

By the way, this is freeware.