Importer Implementation

Importers turn a file's contents into a string that can then be tokenized and indexed.

---

Every importer will take in a filename and then return a string with that file's contents.

In the cases where a single importer has to take in multiple types (such as the doc importer), the importer is passed the filename and file type as a string.

So, calling the text importer may look like this: importer(myFileName.txt)

And calling the doc importer may look like this: importer(myFileName.docx, 'docx')

Any new importer added must have an importer function. Importers usually utilize the subprocess function of python to convert files to plain text. An outside program is called using subprocess.call() that converts whatever file type you are given to plain text, which is then stored in a temporary file.

For example, pdftotext is used to convert text-based pdf files to plain text. So, pdftotext is called using the filename passed to the importer, then it outputs to a temp file. Then, python can read in the temp file and turn it into a string.

---

Temporary files are used for the output reduce clutter of the file system and to avoid reading in previous versions of a file. In the case of some importers, however, regular files are stored inside of a temporary directory. When the function is done, that directory and all of its contents are removed from the system.