RESOLVED - com.ibm.wsspi.webcontainer.util.RequestUtils parseQueryString SRVE0325E: Exceeding maximum parameters allowed per request 10,000 ,current 10,000 , cannot add more.(SOLR)

Sometimes , you may find that the SOLR Search and Browse cease to function and you may keep hitting "No Search Results". On further investigating the SOLR logs , you may find the below exceptions -

[5/26/15 11:27:08:586 BST] 00000027 SolrCore      I org.apache.solr.core.SolrCore execute [MC_10001_CatalogEntry_en_GB] webapp=/solr path=/select params={q=shortDescription:tv} hits=905 status=0 QTime=1

[5/26/15 11:27:36:127 BST] 00000027 util          E com.ibm.wsspi.webcontainer.util.RequestUtils parseQueryString SRVE0325E: Exceeding maximum parameters allowed per request 10,000 ,current 10,000 , cannot add more.

[5/26/15 11:27:36:131 BST] 00000027 SolrDispatchF E org.apache.solr.common.SolrException log java.lang.IllegalArgumentException

        at com.ibm.wsspi.webcontainer.util.RequestUtils.parseQueryString(RequestUtils.java:196)

        at com.ibm.ws.webcontainer.servlet.RequestUtils.parsePostData(RequestUtils.java:356)

        at com.ibm.ws.webcontainer.srt.SRTServletRequest.parseParameters(SRTServletRequest.java:2051)

        at com.ibm.ws.webcontainer.srt.SRTServletRequest.getParameterMap(SRTServletRequest.java:2568)

        at org.apache.solr.request.ServletSolrParams.<init>(ServletSolrParams.java:29)

        at org.apache.solr.servlet.StandardRequestParser.parseParamsAndFillStreams(SolrRequestParsers.java:394)

        at org.apache.solr.servlet.SolrRequestParsers.parse(SolrRequestParsers.java:115)

        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)

        at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190)

        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125)

        at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80)

        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)

        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:935)

        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:503)

        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest

This is caused when the SOLR query string has more than 10,000 query string parameters. SOLR cannot handle that and fails with an internal server error. To resolve this issue , please follow the below steps -

1. In the administrative console click 
Servers > Application Servers > server_name > Web Container settings > Web Container 

2. Under Additional Properties select Custom Properties. And add a new 

Property name: com.ibm.ws.webcontainer.maxParamPerRequest 

Value : some number 

The default value for this is 10000. You can set it a different number or you can set this property to -1 if you do not want to limit the number of parameters that can be included in a request
 

WC Search Customization Points

A simple Mind map which depicts the commonly customized files/components with their purpose while extending the WC Search.

SearchPreProcessor
A query preprocessor modifies the native SolrQuery object right before it is sent to the Solr server for processing.

The custom query preprocessor needs to be mapped in the wc-search.xml.

For more details , refer to the infocenter link

SearchPostProcessor
A query postprocessor modifies the physical DataObject, SolrEntityContainerImpl, immediately after the QueryResponse is returned from the Solr server.

The custom query postprocessor needs to be mapped in the wc-search.xml

For more details , refer the infocenter link 

SearchProfile
WebSphere Commerce search uses search profiles to control the storefront search experience at a page-level. Search profiles group sets of search runtime parameters such as search index name, search index fields, expression providers and result filters, paging and sorting, and search feature configurations such as text highlighting, facets, and spelling correction. Search profiles are defined in the WebSphere Commerce search configuration file (wc-search.xml). 

For more details , refer the infocenter link 

Business Object Mediator
The business object mediator maps physical search index fields to their corresponding logical fields in the CatalogEntryView noun. Custom physical index fields can be mapped by using the UserData fields.

For more details , refer the infocenter link 

Preprocessor and Data Import Handler (DIH)
The preprocessing tasks are controlled by the wc-dataimport-preprocess XML files. The files contain table definitions, database schema metadata, and references to the Java classes used in the preprocessing steps. Those files are invoked by the di-preprocess utility, which extracts and flattens WebSphere Commerce data and then outputs the data into a set of temporary tables inside the WebSphere Commerce database. The data in the temporary tables is then used by the index building utility to populate the data into Solr indexes using Solr's Data Import Handler (DIH). 

For more details , refer the infocenter link

Search Schema
Typically, WebSphere Commerce data is stored in a relational database. In WebSphere Commerce search, another repository also stores WebSphere Commerce data. That is, the product information is stored in a search index for searching purposes. Most of the data that are stored in the search index is extracted from the relational database. Similar to the database schema, the search index also contains a schema. The fields that are defined in this schema are used by WebSphere Commerce search.
By customizing the schema.xml , we define fields in the search schema that store the data to be searched. Once the new fields are defined, the mappings must be configured for the fields to the database schema.

For more details , refer the infocenter link

I hope you will find the mindmap really useful. As always , please feel free to add your comments and suggestions.

Customization of Search Catalog Navigation

An interesting and easy to understand tutorial to customize Search Catalog Navigation from a fellow blogger!

http://wcsconcepts.blogspot.co.uk/2014/07/customising-catalognavigationview-to.html

Happy Searching! :)

WebSphere Commerce Search Cookbook

The IBM Support blog, called the 'Commerce Search Cookbook' covers common customizations requested by customers.

https://www.ibm.com/developerworks/community/blogs/CommerceSearch/?lang=en

This is a great reference  point for SOLR issues and solutions - so sharing with the readers.

Happy Exploring!

WC Search Customizations - Introducing new facets

Faceted search is the dynamic clustering of items or search results into categories that let users drill into search results (or even skip searching entirely) by any value in any field. Each facet displayed also shows the number of hits within the search that match that category. Users can then “drill down” by applying specific contstraints to the search results. Faceted search is also called faceted browsing, faceted navigation, guided navigation and sometimes parametric search.

One of the common customization scenario is to introduce custom attributes as facets. The following should guide a developer through the process:

Important SOLR Configuration Files

Location : <WCDE>\search\solr\home\MC_<master_catalog_id>\en_US\CatalogEntry\conf

1. schema.xml
Introduce a child element "field" for the element "fields" for CATENTRY.FIELD1.

Place it at the end of the attributes under "Catentry's basic attributes" for logical grouping

<fields>
 <!--
   Catentry's basic attributes: map to table CATENTRY
   -->
<field name="catentry_id" type="long" indexed="true" stored="true" required="true" multiValued="false"/>
.
.
<field name="catenttype_id_ntk_cs" type="wc_keywordText" indexed="true" stored="true"  multiValued="false"/> 
 <field name="field1" type="long" indexed="true" stored="true" multiValued="false"/>

where ,
indexed: true if this field should be indexed (searchable or sortable)
stored: true if this field should be retrievable
multiValued: true if this field may contain multiple values per document

2. wc-data-config.xml
We will update the queries that are used to populate the Solr index from the WebSphere Commerce database tables. Each query needs to retrieve the data stored in field1 of the CATENTRY table.The wc-data-config.xml file loads catalog entry data in three separate parts. Product data is loaded first. This is followed by bundle data and finally all other catalog entry types. For each of the three sections there is a query for populating the full index and another query for delta updates. For each section (product, bundle and other), we need to update the two queries and then add a new field element to assign the data from the WebSphere Commerce database to the field1 field in the Solr index. All together there are nine updates you need to make to this file.

The modified file can be downloaded from here. Search "FIELD1" in the downloaded file to know the modifications required.


Important Database Tables
SRCHATTR

SRCHATTRPROP

SQL updates required

insert into srchattr (srchattr_id, indexscope, indextype, identifier) values ((select max(srchattr_id)+1 from srchattr ) , '0', 'CatalogEntry','_cat.Field1');

insert into srchattrprop (srchattr_id, propertyname, propertyvalue)
values ((select max(srchattr_id) from srchattr ),'facet','field1');

Here , propertyvalue ' field1' is the name specified in the schema.xml

Also, srchattr_id should match in both the above queries

Code File Changes



1. SearchFacetsDisplay.jspf
Location : /Stores/WebContent/Madisons/Snippets/ReusableObjects/
This JSP fragment loops through the available global facets and includes another JSP fragment to display each facet’s data.We will introduce code to display the facet data for field1 in the left sidebar.

Search for the below <c:otherwise> tag and introdule the below code before the tag:

<c:when test="${facetField.value eq 'field1'}">
         <%@ include file="../../Snippets/Search/Field1FacetDisplay.jspf" %>
         <c:set var="f" value="${f + 1}" />                   
</c:when>

We will create  "Field1FacetDisplay.jspf" in the next step.

2. Field1FacetDisplay.jspf
Download the file from here and place it in the location "/Stores/WebContent/Madisons/Snippets/Search/".

Republish and Test!

Demo Video
Coming shortly