• Deploy, embed, and integrate Solr with a host of programming languages
• Implement faceting in e-commerce and other sites to summarize and navigate the results of a text search
• Enhance your search by highlighting search results, offering spell-corrections, auto-suggest, finding “similar” records, boosting records and fields for scoring, phonetic matching
• Informative and practical approach to development with fully working examples of integrating a variety of technologies
• Written and tested for Solr 1.4 pre-release 2009.08

Language English
Paperback 336 pages [191mm x 235mm]
Release date August 2009
ISBN 1847195881
ISBN 13 978-1-847195-88-3
Author(s) David Smiley, Eric Pugh
Topics and Technologies Open Source

If you are a developer building a high-traffic web site, you need to have a terrific search engine. Sites like Netflix.com and Zappos.com employ Solr, an open source enterprise search server, which uses and extends the Lucene search library. This is the first book in the market on Solr and it will show you how to optimize your web site for high volume web traffic with full-text search capabilities along with loads of customization options. So, let your users gain a terrific search experience.
This book is a comprehensive reference guide for every feature Solr has to offer. It serves the reader right from initiation to development to deployment. It also comes with complete running examples to demonstrate its use and show how to integrate it with other languages and frameworks.
This book first gives you a quick overview of Solr, and then gradually takes you from basic to advanced features that enhance your search. It starts off by discussing Solr and helping you understand how it fits into your architecture—where all databases and document/web crawlers fall short, and Solr shines. The main part of the book is a thorough exploration of nearly every feature that Solr offers. To keep this interesting and realistic, we use a large open source set of metadata about artists, releases, and tracks courtesy of the MusicBrainz.org project. Using this data as a testing ground for Solr, you will learn how to import this data in various ways from CSV to XML to database access. You will then learn how to search this data in a myriad of ways, including Solr's rich query syntax, "boosting" match scores based on record data and other means, about searching across multiple fields with different boosts, getting facets on the results, auto-complete user queries, spell-correcting searches, highlighting queried text in search results, and so on.
After this thorough tour, we'll demonstrate working examples of integrating a variety of technologies with Solr such as Java, JavaScript, Drupal, Ruby, XSLT, PHP, and Python.
Finally, we'll cover various deployment considerations to include indexing strategies and performance-oriented configuration that will enable you to scale Solr to meet the needs of a high-volume site.

• Blend structured data with real search features
• Import CSV formatted data, XML, common document formats, and from databases
• Deploy Solr and provide reference to Solr's query syntax from the basics to range queries
• Enhance search results with spell-checking, auto-completing queries, highlighting search results, and more.
• Secure Solr
• Integrate a host of technologies with Solr from the server side to client-side JavaScript, to frameworks like Drupal
• Scale Solr using replication, distributed searches, and tuning

The book takes a tutorial approach with fully working examples. It will show you how to implement a Solr-based search engine on your intranet or web site.

This book is for developers who would like to use Solr for their applications. You only need to have basic programming skills to use Solr. Knowledge of Lucene is certainly a bonus.

David Smiley

Born to code, David Smiley is a senior software developer with 10 years of experience in the defense industry using Java and various web technologies. David is a strong believer in the open-source development model and has made small contributions to various projects over the years.

David began using Lucene way back in 2000 and was immediately excited by it and its future potential. Later on he went to use the Lucene-based "Compass" library to construct a very basic search server similar in spirit to Solr. Since then, David has used Solr for a larger search project and was able to contribute modifications back to the Solr community. Although preferring open-source solutions, David has also been trained on the commercial Endeca search platform and is currently using that product as well as Solr for a different project.

Eric Pugh

Fascinated by the "craft" of software development, Eric Pugh has been heavily involved in the open source world as a developer, committer, and user for the past 5 years. He is a member of the Apache Software Foundation and lately has been mulling over how we move from the read/write web to the read/write/share web.

In biotech, financial services, and defense IT, he has helped European and American companies develop coherent strategies for embracing open source software. As a speaker he has advocated the advantages of Agile practices in software development.

Eric became involved in Solr when he submitted the patch SOLR-284 for Parsing Rich Document types such as PDF and MS Office formats that became the single most popular patch as measured by votes! SOLR-284 became part of Solr version 1.4.

Table of Contents

Preface
Chapter 1: Quick Starting Solr

• An introduction to Solr
  • Lucene, the underlying engine
  • Solr, the Server-ization of Lucene

• Comparison to database technology
• Getting started
  • The last official release or fresh code from source control
  • Testing and building Solr
  • Solr's installation directory structure
  • Solr's home directory
  • How Solr finds its home
  • Deploying and running Solr

• A quick tour of Solr!
  • Loading sample data
  • A simple query
  • Some statistics

• The schema and configuration files
• Solr resources outside this book
• Summary

Chapter 2: Schema and Text Analysis

• MusicBrainz.org
• One combined index or multiple indices
  • Problems with using a single combined index

• Schema design
  • Step 1: Determine which searches are going to be powered by Solr
  • Step 2: Determine the entities returned from each search
  • Step 3: Denormalize related data
    • Denormalizing—"one-to-one" associated data
    • Denormalizing—"one-to-many" associated data
  • Step 4: (Optional) Omit the inclusion of fields only used in search results

• The schema.xml file
  • Field types
  • Field options
  • Field definitions
    • Sorting
    • Dynamic fields
    • Using copyField
    • Remaining schema.xml settings

• Text analysis
  • Configuration
  • Experimenting with text analysis
  • Tokenization
  • WorkDelimiterFilterFactory
  • Stemming
  • Synonyms
    • Index-time versus Query-time, and to expand or not
  • Stop words
  • Phonetic sounds-like analysis
  • Partial/Substring indexing
    • N-gramming costs
  • Miscellaneous analyzers

• Summary

Chapter 3: Indexing Data

• Communicating with Solr
  • Direct HTTP or a convenient client API
  • Data streamed remotely or from Solr's filesystem
  • Data formats

• Using curl to interact with Solr
• Remote streaming
• Sending XML to Solr
  • Deleting documents
  • Commit, optimize, and rollback

• Sending CSV to Solr
  • Configuration options

• Direct database and XML import
  • Getting started with DIH
    • The DIH development console
    • DIH documents, entities
    • DIH fields and transformers
  • Importing with DIH

• Indexing documents with Solr Cell
  • Extracting binary content
  • Configuring Solr
  • Extracting karaoke lyrics
  • Indexing richer documents

• Summary

Chapter 4: Basic Searching

• Your first search, a walk-through
• Solr's generic XML structured data representation
• Solr's XML response format
  • Parsing the URL

• Query parameters
  • Parameters affecting the query
  • Result paging
  • Output related parameters
  • Diagnostic query parameters

• Query syntax
  • Matching all the documents
  • Mandatory, prohibited, and optional clauses
    • Boolean operators
  • Sub-expressions (aka sub-queries)
    • Limitations of prohibited clauses in sub-expressions
  • Field qualifier
  • Phrase queries and term proximity
  • Wildcard queries
    • Fuzzy queries
  • Range queries
    • Date math
  • Score boosting
  • Existence (and non-existence) queries
  • Escaping special characters

• Filtering
• Sorting
• Request handlers
• Scoring
  • Query-time and index-time boosting
  • Troubleshooting scoring

• Summary

Chapter 5: Enhanced Searching

• Function queries
  • An example: Scores influenced by a lookupcount
  • Field references
  • Function reference
    • Mathematical primitives
    • Miscellaneous math
    • ord and rord
  • An example with scale() and lookupcount
    • Using logarithms
    • Using inverse reciprocals
    • Using reciprocals and rord with dates
  • Function query tips

• Dismax Solr request handler
  • Lucene's DisjunctionMaxQuery
    • Configuring queried fields and boosts
  • Limited query syntax
  • Boosting: Automatic phrase boosting
    • Configuring automatic phrase boosting
    • Phrase slop configuration
  • Boosting: Boost queries
  • Boosting: Boost functions
  • Min-should-match
    • Basic rules
    • Multiple rules
    • What to choose
  • A default search

• Faceting
  • A quick example: Faceting release types
    • MusicBrainz schema changes
  • Field requirements
  • Types of faceting
  • Faceting text
  • Alphabetic range bucketing (A-C, D-F, and so on)
  • Faceting dates
    • Date facet parameters
  • Faceting on arbitrary queries
  • Excluding filters
    • The solution: Local Params
  • Facet prefixing (term suggest)

• Summary

Chapter 6: Search Components

• About components
• The highlighting component
  • A highlighting example
  • Highlighting configuration

• Query elevation
  • Configuration

• Spell checking
  • Schema configuration
  • Configuration in solrconfig.xml
    • Configuring spellcheckers (dictionaries)
    • Processing of the q parameter
    • Processing of the spellcheck.q parameter
  • Building the dictionary from its source
  • Issuing spellcheck requests
  • Example usage for a mispelled query
    • An alternative approach

• The more-like-this search component
  • Configuration parameters
    • Parameters specific to the MLT search component
    • Parameters specific to the MLT request handler
    • Common MLT parameters
  • MLT results example

• Stats component
  • Configuring the stats component
  • Statistics on track durations

• Field collapsing
  • Configuring field collapsing

• Other components
  • Terms component
  • termVector component
  • LocalSolr component

• Summary

Chapter 7: Deployment

• Implementation methodology
  • Questions to ask

• Installing into a Servlet container
  • Differences between Servlet containers
    • Defining solr.home property

• Logging
  • HTTP server request access logs
  • Solr application logging
    • Configuring logging output
    • Logging to Log4j
    • Jetty startup integration
    • Managing log levels at runtime

• A SearchHandler per search interface
• Solr cores
  • Configuring solr.xml
  • Managing cores
  • Why use multicore

• JMX
  • Starting Solr with JMX
    • Take a walk on the wild side! Use JRuby to extract JMX information

• Securing Solr
  • Limiting server access
    • Controlling JMX access
  • Securing index data
    • Controlling document access
    • Other things to look at

• Summary

Chapter 8: Integrating Solr

• Structure of included examples
  • Inventory of examples

• SolrJ: Simple Java interface
  • Using Heritrix to download artist pages
  • Indexing HTML in Solr
  • SolrJ client API
    • Indexing POJOs
  • When should I use Embedded Solr
    • In-Process streaming
    • Rich clients
    • Upgrading from legacy Lucene

• Using JavaScript to integrate Solr
  • Wait, what about security?
  • Building a Solr powered artists autocomplete widget with jQuery and JSONP
  • SolrJS: JavaScript interface to Solr

• Accessing Solr from PHP applications
  • solr-php-client
  • Drupal options
    • Apache Solr Search integration module
    • Hosted Solr by Acquia

• Ruby on Rails integrations
  • acts_as_solr
    • Setting up MyFaves project
    • Populating MyFaves relational database from Solr
    • Build Solr indexes from relational database
    • Complete MyFaves web site
  • Blacklight OPAC
    • Indexing MusicBrainz data
  • Customizing display
  • solr-ruby versus rsolr

• Summary

Chapter 9: Scaling Solr

• Tuning complex systems
  • Using Amazon EC2 to practice tuning
    • Firing up Solr on Amazon EC2

• Optimizing a single Solr server (Scale High)
  • JVM configuration
  • HTTP caching
  • Solr caching
    • Tuning caches
  • Schema design considerations
  • Indexing strategies
    • Disable unique document checking
    • Commit/optimize factors
  • Enhancing faceting performance
  • Using term vectors
  • Improving phrase search performance
    • The solution: Shingling

• Moving to multiple Solr servers (Scale Wide)
  • Script versus Java replication
  • Starting multiple Solr servers
    • Configuring replication
  • Distributing searches across slaves
    • Indexing into the master server
    • Configuring slaves
  • Distributing search queries across slaves
  • Sharding indexes
    • Assigning documents to shards
    • Searching across shards

• Combining replication and sharding (Scale Deep)
• Summary

1. KM & ECM Conference Fall 2009
주    제
경제에 활기를! 기업에 비전을!

일    시
2009년 9월 24일(목) 9:00 ~ 17:30

장    소
삼성동 코엑스 그랜드볼룸(봉은사 맞은편)

행사목적
- KMS, ECM, EP/EIP, EDMS, Search Engine 관련 시장 활성화 및 성공사례 전파
- 정부, 공공기관, 금융기관 및 각 기업체의 생산성 향상과 경쟁력 강화
- 국내외 KMS/ECM 관련 솔루션 최근 동향 및 신기술 관련 최신 정보 제공

행사규모
2트랙 10세션, 전시 15부스 내외, 참관객 500여명

행사내용
- 세미나 : KMS/EDMS/EP/ECM/자료관/기록물관리 관련 사례 및 신기술 발표
- 전시회 : KMS/EDMS/EP/ECM/자료관/기록물관리 관련 제품 전시

참석대상
- 정부/공공/금융 및 일반기업체 전산 부서장 및 담당자
- 각 기업 CEO/CIO/기획실/혁신팀/지식경영 부서장 및 담당자
- 경영혁신의 일환으로 지식경영을 도입하였거나 계획이 있는
  정부/공공기관 및 기업체 관계자
- KMS/ECM을 사용 중이거나 계획이 있는 정부/공공기관 및 기업 관계자

주    최
전자신문사, 한국소프트웨어산업협회

주    관
KM&ECM협의회

참 가 사
가온아이, 굿센테크날러지, 나눔기술, 날리지큐브, 사이버다임, 삼성SDS, 솔트룩스, 아이디에스앤트러스트, 온더아이티, 와이즈소프트

2. [HCLT2009] 한글 및 한국어 정보처리 학술대회 
10월 9일 (장소 : 대전 유성 레전드호텔)

12:30 - 14:00 등록인지과학회 회장 환영사
13:30 - 14:00 한국정보과학회, 언어공학연구회 위원장 인사말, 조직위원장 보고
14:00 - 14:40 초청강연1
14:40 - 15:20 초청강연2
15:20 - 15:40 휴식
15:40 - 17:00 구두발표 세션1(15분 발표 5분 질의응답 ) 4편
17:00 - 17:10 휴식
17:10 - 18:30 구두발표 세션2(15분 발표 5분 질의응답) 4편
18:30 - 20:00 만찬

10월 10일 (장소 : 대전 유성 레전드호텔)

09:00 - 10:20 구두발표 세션3(15분 발표 5분 질의응답) 4편
10:20 - 11:10 포스터 세션
11:10 - 12:30 구두발표 세션4(15분 발표 5분 질의응답) 4편
12:30 - 13:30 점심식사

------------
KM&ECM행사는 솔트룩스를 제외한 검색관련 업체가 빠져서 완전히 KM&ECM전문업체 중심으로 진행이 될 것 같으며,
행사비(사전등록 : 55,000원)가 있어서 초대장이 있지 않으면 참석하는 것이 조금 어려울 것 같다.

HCLT2009행사는 매년 열리는 학회이나 예전에 비해 규모면에서 점점 줄어가는 느낌은 어쩔 수 없다.
공과계열이 천대받는다는 느낌은 여기서도 느낄 수 있는 듯...

1. 전체느낌

- 행사 자체는 비교적 깔끔하게 진행이 되었다.
- 신문기사에는 일천명이 넘는 참관객이 왔다고 하였으나, 작년에 비해 10%~15% 정도 줄은 것 같다. –> 대략적인 느낌임
- 주관사, 참여사부스 간에 모여드는 사람의 수가 확연하게 차이가 났다 –> 이유는 잘 모르겠음
- 쓰리웨어가 처음에 행사하다고 할때는 참관사로 있었는데 실제로는 빠졌다.  --> 오픈베이스는 이미 빠졌었음
- 예상한대로 너무 대놓고 자사의 제품을 홍보하는 발표가 몇몇 눈이 띄었다. –> 특히 참관사..
- 대략적으로
             시맨틱 기술                : 3편
             개인화                       : 1편
             검색UX와 기술의 조합  : 1편
             텍스트 마이닝             : 1편
             검색품질 기술             : 1편
             멀티미디어 검색          : 1편 
             걍~ 제품홍보               : 3편  --> 기술적으로 새로운 시도/개념의 제품도 아니었다.

2. 세션별 느낌

• Keynote – 검색의 미래, 디스커버리 Semantics for Searching and Mining – What, Where, and How? - 맹성현 교수/KAIST 
      참석하였음.
      세미나의 슬로건(“디스커버리! 검색의 미래를 만나다”)에 가장 적합한 내용이 아니었을까?
      semantics기술을 3가지 타입으로 나누고 각각에 대한 설명과 예를 보여주었다.
  
• 시맨틱 검색과 시맨틱 분석의 실용 기술 구현 – 이경일 대표 / 솔트룩스
      참석하지 않음
      솔트룩스의 전반적인 기술적 어프로치를 설명
      자료상으로는 TopicRank를 이끌어 내는 과정과 그후의 내용은 약간 괴리감이 존재한다는 느낌
      TopicRank를 이용해서 얼마나 검색품질을 끌어올릴 수 있었는지에 대한 통계데이타가 있었으면 더 좋지 않았을까?
      참고로 솔트룩스 부스에서 아울림이 보이지 않았던 것은 약간 의외
  
• 사용자 모델링을 이용한 검색서비스 구축전략 – 정휘웅 차장 / 다이퀘스트
      참석하지 않음
      쇼핑몰 강자답게 쇼핑몰을 예제로 검색UX 및 검색기술의 융화방법론을 설명
      검색기술에 관련된 내용은 기존 업체들일 이미 몇년전부터 이야기 하던 것들이라 신선하지는 않았음
      대신 검색UX와 관련된 접근은 다이퀘스트가 아니면 고민하기 힘든 부분이어서 신선했음
  
• 감성검색과 텍스트 마이닝 – 이상주 박사 / 다음소프트
      참석하였음
      감성(Sentiment)에 대한 검색방법을 설명
      예전부터 온라인 풍문에 관한 프로젝트를 많이 하여서인지 데모 프로그램의 전체적인 짜임새는 거의 완벽
      기술적인 부분 이외에 매직(????)을 많이 사용하였다는 생각은 지울 수가 없음
      하지만, 매직도 기술이고 노력이니 인정...
     
• 프로파일링을 이용한 검색의 개인화 – 윤진섭 부장 / 마이크로소프트-패스트
      참석하지 않음
      4~5년전에 Verity, 작년에 오픈베이스가 내놓은 개인화검색과 같은 기술에 대한 설명
      이미 개인화에 대한 시장의 반응은 나온것 같은데...
  
• 멀티미디어 환경에서 구현한 차세대 내용 기반 검색 – 박만수 박사 / 코난테크놀로지
      참석하지 않음
      코난테크놀로지에서 몇년전부터 밀고 있는 멀티미디어 검색에 대한 설명
      자료상으로는 기술이 쏙~ 빠지고 주로 사례를 중심으로 설명하여 약간 김이 샌듯한 느낌
  
• Proximity Language Model – 윤여걸 이사 / 와이즈넛
      참석하였음
      SIGIR ‘09에 발표하였던 검색품질(검색랭킹)을 개선하는 방법에 대한 논문을 설명
      문서내에서 쿼리단어가 얼마나 가깝게, 자주 나타나는지를 계산하여 검색랭킹에 반영하는 방법
      세미나를 시작하기전엔 사람들이 확~ 빠져나가서 반정도 밖에 없었음 –> 역시 학술전문 세미나는 관심이 떨어짐
     
• Enterprise Social Network을 활용한 Knowledge Discovery – 김승연 차장 / IBM
      참석하지 않음
      IBM의 SNF(Social Network Framework)제품을 소개
      단순하게 이야기하면 기관(혹은 회사)내에서 구성원들간의 Action(댓글달기, 메일보내기, 회의참석 등)을 분석하여
      사람들간의 관계 및 어떤 분야의 전문가를 찾아주는 시스템
      간단하게 무슨 효과가 있을지를 생각할 수도 있는데, 규모가 아주 큰 회사에서는 적당한 사람을 찾는 것 또한 굉장한 리소스가 들게 마련이다.
      부스에서 간단하게 시연을 구경하였는데 그다지 감흥은 없었다는...
     
• 아웃링크 버티컬통합검색 개발에 사용된 차세대 검색기술 소개 – 류홍진 대표이사 / 건지소프트
      참석하지 않음
      노 코맨트
  
• 시맨틱 정보서비스의 진화 – 정한민박사 / KISTI
      참석하였음
      온톨로지와 미래의 인터넷에 대한 설명
      개인적으로 Web 3.0을 논하는걸 좋아하지 않지만, Mashup에 관련된 부분은 공감이 감
      참고로 Ontoframe에 대한 설명를 다이퀘스트 부스에서 하고 있었던 것 같음
  
• 강력한 문서필터엔진 오라클 Outside-in – 장경운 부장 / Oracle
      참석하였음
      오라클에서 출시한 문서필터 제품 Outside-in이란 제품 홍보
      한글파일까지 필터링되는것은 와우~ 
      하지만, 삼성그룹에서 사용하는 훈민정음, 정음글로벌 문서는 필터링이 않된다는…
      가격이 사이냅보다 매력적일까?

1. 주관사가 생겼다.
       아무래도 업무진행을 원활하게 하기 위해서 일 것이다.
2. 주최사에서 오픈베이스가 빠지고 다음소프트가 들어갔다.
       쓰리소프트가 빠진것은 알겠는데… 오픈베이스도 빠졌다.
       예상컨데 비용때문은 아니 것 같고 다른 문제가 있으리라…
3. 주최사 이외에 참여사가 생다.
       컨퍼런스 자체가 무료로 진행되기 때문에 주최사쪽에서는 아무래도 비용이 문제가 된다.
       때문에, 참여사를 늘려 비용을 낮추려 했으리라…
4. 교수님들 발표비중이 확~ 줄고 대신 참여사 발표가 생겼다.
       이 부분에서 작년의 취지와는 약간 달라진 듯한 느낌이 드는건 어쩔 수 없다.
       물론 작년에도 교수님들 초빙을 업체에서 했기 때문에 업체의 입김(?)에 완전히 자유롭지는 못하였지만
       올해는 아예 대 놓고 회사/제품 홍보(?)하겠다는 것이다.
       때문에, 올해는 대학생들 보다는 실제 업무에 적용해야 하는 직장인들이 더 유용한 컨퍼런스가 될 듯 하다.
   
   뭐… 어째튼 나는 가능하면 참석할 예정이다.
   물론, 가서 만나는 사람은 별로 없겠지만 어떤 제품과 기능들을 홍보하는지 알아보는것도 큰 공부가 되니까…

출처의 내용은 최악의 경기불안 요인에도 불구하고 ‘코난테크놀로지’, ‘코리아와이즈넛’은 최고의 한해를 보내고 있다는 것이다.
물론 기사의 내용에 나와 있듯이 내실경영, 정부의 상반기 조기발주 등의 원인도 있을 것이나 기사에 나오지 않는 다른 원인도 있을 것이리라…

작년까지 기업검색 시장은 3강체제를 유지하고 있었다.
하지만, 한축을 담당하던 쓰리소프트가 갑작스럽게(?) 사업을 철수하게 되어 자연스럽게 다른 두 회사가 이득을 볼 수밖에 없지 않았을까?
물론, 쓰리소프트의 자원을 쓰리웨어가 가져갔다고는 하나 규모나 인지도면에서 떨어질 수 밖에 없다고 할 수 밖에 있다.

예상하건데, 이런 체제(?)는 향후 몇년간은 이어질 것으로 판단되며 이를 바탕으로 사업적인 전략을 구상해야 할 것으로 판단된다.

솔트룩스가 잘~ 할수 있는 몇몇 기능들이 눈에 보이고 여기저기 신경을 많이 쓴 표시가 난다.
전체적으로는 잘 구성된 시스템으로 보인다. (물론 실제로 사용해봐야 정확하게 알 수 있겠지만…)

1. 텍스트마이닝 기능들(군집, 요약, 추출, 유사문서검색)은 장점으로 이야기 할 수 있을 것 같다.
2. 하드웨어 일체형으로 비용, 관리, 설치를 쉽게 할 수 있도록 하고 있다.
3. 얼마나 사용될지는 모르겠지만, 문서내의 “이미지”, “그래프”, “표”등의 오브젝트를 검색할 수 있는 기능이 있다.
4. 문서이력, 버전관리 기능이 있다.

몇가지 생각할 점……

1. 아무래도 공유폴더 검색의 핵심은 ACL(문서접근권한)이 될 것 같다.
   하지만, “서치박스”의 기능 설명으로 보면 “문서다운로드시” 사용자권한을 체크한다고 되어 있는데…
   이는 검색시에는 권한이 없는 사용자일지라도 모든 문서를 검색할 수 있다는 얘기가 된다.
   이는, 보안에는 좀 치명적이지 않을까?
   기술적인 어려움은 알겠는데 어렵다고 넘길 부분은 아닌 것 같다.
2. 일반적으로 “통합검색”을 구현할 때 “파일서버”도 같이 검색될 수 있도록 구현되지 않나?
   따라서, “파일서버”만 별도로 검색되어지는 시스템은 아무래도 시장성이 떨어질 것 같다는…
3. 일반 데스크탑검색 패키지로도 공유문서 검색같은 것은 구현할 수 있지 않을까?
4. 약 2년전부터 패키지 형태로 유사제품을 판매하고 있는 K업체에 비해 시장진입이 너무 늦은 것 아닌것 아닌가?

해외에도 이와 비슷한 그룹(?)이 있지 않을까?

있다.

search_dev ( Independent Search Engine Developers)

irgroup과 다른점은 웹사이트가 아니고 뉴스그룹이라는 점과, irgroup이 검색엔진 개발에 촛점이 맞추어져 있다면 search_dev는 주요 벤더의 검색엔진 활용에 촛점이 맞춰져 있다는 점이다.

상당히 열성적인 토론이 이루어지고 있으니 관심있으신 분들은 방문해 보시길...

관심링크 : Japanese search in autonomy

코난, SK계열 검색사업 잇단 수주
--> 예견된 일

넷킬러, 기업용 검색도구 '구글 미니' 공급
--> 잘 될까?

파이썬 기반, Open Source search engine : Xappy
--> 또 하나의 Open Soruce 검색엔진의 출현이군요...

우리나라의 많은 검색엔진들은 사이냅소프트의 필터 모듈을 정기계약 형식으로 사용하고 있다.
포탈검색이나 기업검색 모두에서 오랜시간동안 꽤 안정적인 성능을 보여주고 있다고 할 수 있다.

Open Source쪽에서도 이러한 모듈이 당연히 필요하지 않겠는가?

그래서 Apache Lucene의 sub project로 Tika가 개발되고 있으며, 현재 0.2버전까지 나온 상황이다.
(어찌보면) 당연하게 한글, 훈민정흠, 정음글로벌과 같은 문서 포맷은 지원 되지 않지만, Lucene이나 Solr을 사용하는 상황에서 MS계열의 문서들이나 PDF문서들만을 필터링하려고 하면 고려해 볼만 하지 않을까?

현재 지원되는 문서포맷
-------------

Microsoft's OLE 2 Compound Document format

A number of Microsoft applications, most notably the Microsoft Office suite, use the generic OLE 2 Compound Document format as the basis of their document formats. Tika uses Apache POI to support a number of these formats.

The OLE2 Compound Document format is designed for use with random access files, and so the input stream passed to a Tika parser needs to be spooled in memory or in a temporary file depending on the size of the document. See TIKA-153 for an effort to avoid this extra temporary file if the input document already comes from a file.

In addition to the shared base format there's also a shared sets of metadata in typical OLE2 documents. Tika uses the HPSF library from POI to parse these property sets and exposes them as the following document metadata:

• TITLE Title
• SUBJECT Subject
• AUTHOR Author
• KEYWORDS Keywords
• COMMENTS Comments
• TEMPLATE Template
• LAST_SAVED Last Saved By
• REVISION_NUMBER Revision Number
• LAST_PRINTED Last Printed
• LAST_SAVED Last Saved Time/Date
• LAST_SAVED Last Saved Time/Date
• PAGE_COUNT Number of Pages
• WORD_COUNT Number of Words
• CHARACTER_COUNT Number of Characters
• APPLICATION_NAME Name of Creating Application

Note that in practice the metadata in many documents is either missing, incomplete or even incorrect, so a client application should not rely too much on this information.

Support for the new Office Open XML format used by Microsoft Office version 2007 is pending for a POI upgrade. Current status is recorded in TIKA-152 .

The generic OLE2 Compound Document format is automatically detected using a magic number, and further parsing can automatically determine the more specific document format. Tika also knows a number of common glob patterns like *.doc and *.ppt for these formats.

The supported OLE 2 Compound Document formats are:

Microsoft Excel (application/vnd.ms-excel)

Excel spreadsheet support is available in all versions of Tika and is based on the HSSF library from POI.

The Excel parser in Tika uses the HSSF event API and is able to extract much of the document structure, including all (non-empty) worksheets and their table structures. Formula results are extracted as stored in the Excel file, and cell links are exposed as XHTML links. These features were added in Tika version 0.2.

Cell comments and formatting are currently not supported. See TIKA-148 and TIKA-103 for the respective issues.

See the ExcelParserTest test case for an example of parsing Microsoft Excel files.

Microsoft Word (application/msword)

Word document support is available in all versions of Tika and is based on the HWPF library from POI.

The Word parser uses the WordExtractor class from HWPF to extract document content as a sequence of paragraphs.

See the WordParserTest test case for an example of parsing Microsoft Word files.

Microsoft PowerPoint (application/vnd.ms-powerpoint)

PowerPoint presentation support is available in all versions of Tika and is based on the HSLF library from POI.

The PowerPoint parser uses the PowerPointExtractor class from HSLF to extract spreadsheet content as a single paragraph.

See the PowerPointParserTest test case for an example of parsing Microsoft PowerPoint files.

Microsoft Visio (application/vnd.visio)

Visio diagram support was added in Tika version 0.2 and is based on the HDGF library from POI.

The Visio parser uses the VisioExtractor class from HDGF to extract diagram content as a sequence of paragraphs.

Microsoft Outlook (application/vnd.ms-outlook)

Outlook message support was added in Tika version 0.2 and is based on the HSMF library from POI.

The Outlook parser extracts the subject of the message and the From, To, Cc, and Bcc addresses (formatted for display) along with the body text of text/plain messages. The AUTHOR , TITLE and SUBJECT metadata properties are set explicitly, overriding potential generic document metadata retrieved from OLE2 property sets.

Compression formats

General purpose compression formats are used to reduce the size of any kinds of documents. Tika uses a parsing pipeline to support general purpose compression: in the first stage the compressed stream decompressed and the resulting decompressed stream is passed on to a second parsing stage where it will be processed as if the document had never been compressed.

Tika contains magic numbers and glob patterns for auto-detecting all supported compression formats. The glob patterns of compression formats are also used to determine the name of the original uncompressed document. If a client application has supplied a RESOURCE_NAME_KEY metadata property that matches such a glob pattern, then the decompressing first parsing stage will replace the RESOURCE_NAME_KEY metadata property with the deduced original document name before passing control to the second parsing stage.

Note that apart from the special handling of the RESOURCE_NAME_KEY property, no document metadata is passed to or from the second parsing stage. Only the text content extracted by the second stage parser is returned to the client application.

The supported compression formats are:

gzip compression (application/x-gzip)

Gzip support was added in Tika version 0.2 and is based on the GZIPInputStream class in the Java 5 class library.

The known gzip glob patterns are *.tgz , *.gz and *-gz , and they will respectively be replaced with *.tar , * and * as described above.

bzip2 compression (application/x-bzip)

Bzip2 support was added in Tika version 0.2 and is based on bzip2 parsing code from Apache Ant , which in turn was originally based on work by Keiron Liddle from Aftex Software.

The known bzip2 glob patterns are *.tbz , *.tbz2 , *.bz and *.bz2 , and they will respectively be replaced with *.tar , *.tar , * and * as described above.

Other supported formats

Extensible Markup Language (application/xml)

Tika uses the javax.xml classes to parse Extensible Markup Language files. Support for Extensible Markup Language files was added in Tika 0.1.

HyperText Markup Language (text/html)

Tika uses the CyberNeko library to parse HyperText Markup Language files. Support for HyperText Markup Language files was added in Tika 0.1.

Images (image/*)

Tika uses the javax.imageio classes to extract Metadata from Image files. Support for Image files was added in Tika 0.2.

Java class files

The parsing of Java Class files is based on the asm library and work by Dave Brosius in JCR-1522. Support for Java Class files was added in Tika 0.2.

Java jar archives

The parsing of Java JAR archives is performed using a combination of the ZIP and Java class file parsers. Support for Java JAR archives was added in Tika 0.2.

MP3 Audio (audio/mp3)

The parsing of ID3v1 tags from MP3 files was added in Tika version 0.2. If found the following metadata is extracted and set:

• TITLE Title
• SUBJECT Subject

The above information, as well as the Album , Track , Year , Genre and additional Comment are extracted when set in the file.

OpenDocument (application/vnd.oasis.opendocument.*)

TODO

Plain text (text/plain)

Tika uses the International Components for Unicode Java library (ICU4J) to parse plain text. Support for plain text was added in Tika 0.1.

Extracting text content from plain text files is actually a relatively complex task due to the fact that the character encoding of the text file is often unknown to the parser.

The text parser in Tika uses the ICU4J CharsetDetector class to automatically detect the character encoding of any text input. As an added benefit, the ICU4J library is in some cases able to detect also the language in which the text is written.

The character encoding and language of the plain text document are returned as the Metadata.CONTENT_ENCODING and Metadata.LANGUAGE metadata properties. If the (declared) content encoding of a text document is already known to the client application, then it can be supplied as the Metadata.CONTENT_ENCODING metadata property to the parser to simplify encoding detection.

Portable Document Format (application/pdf)

Tika uses the PDFBox library to parse Portable Document Format (PDF) documents. Support for PDF was added in Tika 0.1.

Rich Text Format (application/rtf)

Tika uses Java's built-in Swing library to parse Rich Text Format (RTF) documents. Support for RTF was added in Tika 0.1.

The RTF parser in Tika uses the Swing RTFEditorKit class to extract all text from an RTF document as a single paragraph. Document metadata extraction is currently not supported.

tar archive (application/x-tar)

Tika uses an adapted version of the tar parsing code from Apache Ant to parse tar archives. The tar code is originally based on work by Timothy Gerard Endres. Support for tar archives was added in Tika 0.2.

ZIP archive (application/zip)

Tika uses Java's built-in Zip classes to parse ZIP files. Support for ZIP was added in Tika 0.2.