Dear Mr. Navarro,

From March 21 to 30 I was honored to be named by you as a local expert supporting the methodological part of your recent sociological survey published at 03.21.2012.

First of all, I want to express my gratitude for regarding me as an expert or a trustworthy person, which is very comforting for every child living inside us. We all come from childhood, as Saint-Exupéry put it. But as an adult and a social scientist, it is surprising for me why you should be quoting me when I have never been aware, consulted or been invited to any meetings concerning research, surveys and polling activities of your institution.

The only meeting and debate between us took place in fall 2011 at the conference organized by your organization concerning gender equality; during which, to my regret, I had to refresh your mind about one of central priorities identified by US Secretary of State, Mrs. Hilary Clinton during her visit to Georgia – increasing women’s involvement in politics and civil activities as one of the central indicators of democratic development. The reason of my statement was your willingness to assert the attitude of members of Georgian parliament, when they stated proudly that gender equality is not a state priority. I asked you a question: whose interests is your organization representing:

• nonprofit, nonpartisan organization working to support and strengthen democratic institutions worldwide through citizen participation, openness and accountability in government [http://www.ndi.org/]

• US official politics gender equality and women’s involvement in building democratic institutions or

• Georgian parliament, which is only trying to formally create an impression that they address the issue of gender equality.

Unfortunately, you never responded to me directly and our debates did not continue due to the conference format. However it was delightful for me that in your current survey there were several questions concerning the gender equality issues.

I do not know why but suddenly twice I appeared to be the person you claimed as a supporter the methodology of your survey on TV channel “Kavkasia”. It was even more of a surprise when you did it the second time on March 30, after I interfered in live streaming on March 24 during your visit at TV channel “Kavkasia” and denied my support and participation regarding the methodological part of your survey.

Since you twice repeated that I support your methodology, which is not true, I decided to write this open letter and spread it via printed media. Since, as an expert of Georgia you will no doubt agree that this open letter will not be distributed by the three nationwide TV channels [based on your survey data from different waves] and the second source of distribution of information is printed media that has about 30% of customers who actually get it late.

Have you ever thought why people do not afford newspapers and coverage is low? Unfortunately your research says nothing about inequality of access to free media in Georgia. As far as I remember the definition of democracy includes in itself free access to information. As it seems there is another definition of democracy developed somewhere else, by which it is much more important to clarify how effectively and efficiently the government works than to identify public attitudes towards the essential principles of democracy [justice, equality, accessibility, accountability] and issue of their protection.

Not to be boring and get back to my concern, I will be simple. Below you can find my arguments about why I will not and cannot bear a responsibility concerning the accuracy of your research methodology. I have no access either to the questionnaire or the database, which would let me engage in more sophisticated discussion.

I categorically disapprove of your way of distributing information on your research, when you do not spread the political party rankings publically, referring them as confidential, and afterwards you do not react to violations of confidentiality from the party side.

I have not read any research that would support your methods of dissemination of the information or the survey, research results, i.e. that the dissemination of parts of information would support either developments of parties or democracy as such. Furthermore, as far as I am aware, the publicity and accessibility of information is one of the essential measurement indicators in the democracy index; besides, the main attribute of the free world is the freedom of information and accessibility. I would be very grateful if you would share other research on this question, if you have any, that will justify your approach as most valid and adequate for democracy support and development.

You declare that I have no doubts about the methodological part of the research, and then discuss the Kish Grid in details.

I would like to remind you that an American scientist, Lawrence Neuman (2003) identifies 6 main stages of public opinion survey: at the I stage, researcher identifies a research hypothesis and inventory, which will be used for data collection – it can be either structured interview or questionnaire; at the II stage pilot study is conducted for testing the inventory; III stage is dedicated to sampling; IV stage is data collection; the V stage data entry and preparation of data base; the final, VI stage is report writing in different formats (papers and analytical reports) that social science is familiar with. [for your information – this is taught on undergraduate level at TSU, within the qualitative research methods course].

Your argument was that, since I support the sampling method [III stage of public opinion survey] and have nothing against Kish methodology of sampling, this automatically means that I support you research methodology. In cognitive psychology this is called cognitive error, arbitrary inference, or overgeneralization.

It would be at least foolish or complete unprofessionalism from the side of a social scientist to doubt the sampling method in social sciences [classic at the moment] identified and established by Leslie Kish in 1965.

Yes, I agree that public opinion survey should be, and in many cases are, conducted via random sampling. I also agree that sampling of 3000 people is representative for Georgia,

But does it also mean that I agree that the method you chose corresponds the research hypothesis, whether you composed the questionnaire correctly, or revised the issues raising identified after the pilot stage, or conducted the interviews accurately and did right supervision for field work, confirmed or denied your hypothesis with the statistical analysis, or whether the report you presented is adequate, reliable and analytically robust to the research you conducted????

In case I was not clear, I state directly, that you speculate on and misinterpret the concept of research methodology. You over generalize saying that a someone supporting one component of methodology frame automatically supports everything, i.e. other 5 components of survey methodology. And these 5 components are essential parts of the research and the reason why the research is conducted at all – in other words, they are the results and the contents of the research. Defending the Kish Grid will not guarantee what the results would be. The answers on the questions survey puts make the result of the research. Consequently the problem is whether the question is adequate or not. I have no idea about your questionnaire [not published]. I would be grateful if you let me know what exactly I am supporting.

Now, concerning what you should have included in methodological part of the research, but did not. In case of public opinion survey, the standard form is filled indicating the following: average length of the interview, language of the interview, confidentiality of the research, training of the interviewers and supervisors, principles of sampling households and respondents, non response, reasons of not responding, distribution of non response in the sample and by geographical area, obstacles to the research, problematic questions, data entry and data weighting, and standard, short annotations on every geographical area according to sampling places. This is a guideline for a standard research, which accompanies none of the surveys of you are in charge, or it does but is not published.

What do I agree with you on? Things I have never seen? Or agreeing on Kish Grid means agreeing on everything?

Let’s speak on a simple example from your survey. Obviously from the published part, I do not comment things you regard as confidential. To be honest, I do not even consider them seriously, since I am one of those social scientists who believe that political rankings are unstable social variables and that using them for forecasting is a matter of political speculation and they are not for serious and calculated social indicators for forecasting or modeling the future development [of course I do not mean modeled chronicles of “Imedi”]. I even suspect that you do not care about the real contents of the research and semantic results. If you did, you would publicly disseminate them, or, operating on your own logic, you would at least protest the breach of confidentiality. In fact, in the US you would sue for violating the agreement and causing moral damage to you and to the reputation of your organization].

Why should not I trust your survey? Because there are logical discrepancies in the tables you presented and generally researchers would be interested in the reasons of these discrepancies – ones should be seeing them, others – explaining the misunderstandings.

For example, from the first part of your research concerning national and local issues we learn that central problem for the majority of the population is unemployment [62%], 79% of the population think that prices have increased; people report that the situation concerning the issues they are concerned about has worsened: 53% of respondents believe the politicians say very little about inflation and 47% believe politicians say very little about poverty. The Georgian population believes the situation has deteriorated because of the increased costs of public transport (64%), utilities (56%), emergency services (45%), and kindergartens (37%); 70% of respondents report themselves as unemployed; and despite of all aforementioned, 55% believe that Georgia is developing in the right direction!!!

Has it not ever occurred to you that this Georgia [since the research is representative] has a serious problem of mental development or there is something else wrong and you can double-check the data using crosstabs. In other words, exclude those people having such kind of cognitive dissonance by connecting two variables and see what percentage they compile. In case of interest one can study the group of people who tend to choose socially desirable responses on politically sensitive questions.

Only then you would have a real picture. That is why I believe that not only your interpretation of the research should be published, but the passport of survey, questionnaire and databases as well, so all of us could double-check what you say, moreover, test the hypothesis we are interested in using these data.

I cannot trust your research since you do not care about the questions included in your own report, the interesting ones; you do not give us any information about who the people are that constitute a certain social group, what their demands, attitudes are, etc.

I can continue without an end, but I will suggest only one small example and will finish with that. The part of the research concerning democracy and reforms contains two interesting pieces of data: only 55% of population speaks about politics, out of which 84% does it at home. The next slide shows that 49% believes that there is democracy in the country, which is followed a slide saying that for 58% of respondents democracy means freedom of speech, and only 17% thinks democracy means free and fair elections.

Pardon me, but we have to stop here. The situation here is much more complicated than in the previous case. If 49% of Georgian population believes that Georgia is a democratic country, for 58% democracy means freedom of speech and it means fair elections for only 17%, it indicates that either we do not know what democracy is or think that democracy can be established only at home, as only there [at home] we speak about politics [out of 55% of people generally speaking about politics, 84% speaks about it at home?????]. Does it say nothing to you?! This has nothing to do with the data of 2011 democracy index, according to which Georgia is a hybrid regime and occupies a 102th place in the world between Cambodia [101] and Kenya [103] {http://en.wikipedia.org/wiki/Democracy_Index }; in this case there is a lack of public space for discussing politics and the society does not and cannot choose any other place but home.

Besides, according to your data, freedom of speech, as a direct indicator of democracy, strangely coincides/confirms the “Freedom House” 2011 report data about stabilization and liberalization of media in Georgia with very high percentage (http://www.freedomhouse.org/report/freedom-world/2011/georgia). And this happens when starting from fall 2011 every single foreign official speaks about fair and free elections. How can I believe that Georgian population regards freedom of speech as a major indicator of democracy and not fair and democratic elections? Maybe this is true, but why should I trust your conclusions, when they are based only on frequency tables and do not use even a single, at least one step upgraded descriptive statistical operation.

Please explain, whether or not there are too many coincidences and whether or not your data raises the assertion that the research is not conducted correctly? If an interviewer approaches a respondent [following Kish Grid] and asks whether he/she is a human being, what response he/she would get? Probably 99% response would be positive. Your questionnaire results also surprisingly repeat the pleasing indices and give us no clue why the population of poor on the one hand, and hybrid-regime country, on the other, give inconsistent and confusing responses.

I want to know why you are accusing me of defending the methodology giving you opportunity to present a report with so many discrepancies.

And finally, there is a sign on your report that the research has been conducted by the Caucasus Research Resource Centers. It gives an impression that this is an independent organization, which is incorrect. Caucasus Research Resource Centers is a name of a program which belongs to Eurasia Partnership Foundation and does not have any right to be represented as an independent organization. I wonder why you do not put the name of Eurasia Partnership Foundation as your contractor and why do not report that the research is conducted by the program of this foundation. I also wonder on what bases you and the mentioned organization are working together – on the bases of service contract or a grant. Why cannot we find this information in the survey summary of the research along with the donor name?

I have a right to ask this as an Advisory Board member of Caucasus Research Resource Centers. The program is founded by Carnegie Endowment on request of which an Advisory Board was created as an entity carrying purely representative functions in order not to lose a connection with local academic network. The program serves or should serve the development of social research standard and quality. The research financed by Carnegie is called Caucasus Barometer and the personnel involved with the program get a fixed salary guaranteed by the Carnegie grant. I want to know whether the research they conducted on your request is a voluntary job, goes under the grant or a service contract.

This is important, as using the name of this program for scandalous and nonscientific [at least applied social science surveys and polling] purposes cannot serve positively the Carnegie objectives – to support the development of evidence based social and political research in Georgia.

Of course, it is not your fault that at CRRC researchers keep silent [not to say a word about the contractor organization and the head of the aforementioned program]. Their silence is a matter of solely their responsibility, but considering that even Roman slaveholders would envy the labor code of Georgia, one can feel completely safe: no one will ever ask, no one will ever explain and the loss is minimal.

In the end, once again I want to express my gratitude for referring me as an expert; as a representative of social science academics network I felt obliged to state publicly my opinion about your research and my concerns towards it, why I cannot accept, share and trust your reports.

It is truly regretful that while writing this letter I had to look only in one book for checking my argumentation – L. Tsuladze, Quantitative Research Methods in Social Sciences, Center for Social Sciences, 2008. This is an additional reading for undergraduate students. It is available for public use and is uploaded on the web-page of Center for Social Sciences. [http://www.ucss.ge/publication/publications ]

I wish you success in your future research and looking forward to more adequate and analytically interesting report from a nonprofit, nonpartisan organization working to support and strengthen democratic institutions worldwide through citizen participation, openness and accountability in government.

 

Sincerely Yours,

Marine Chitashvili

 

TSU Professor of Psychology

Director of Center for Social Sciences

Life Visiting Fellow at Wolfson College, University of Cambridge

 

P.S. The letter was written and translated when the news denying the citizenship to major opposition figure by the civil registry accompanied by the statements of governmental and non-governmental figures justifying the rightness of their decision was disseminated without any equal access for another party expressing his views on the issue by the nationwide media channels to Georgians 70% of whom supported reinstated of citizenship, 70% of whom considers themselves unemployed, 55% believe that Georgia is developing in the right direction and 17% considering the democracy means free and fair elections [!!!!!!].