Teacher Response to Module #5: Hope We’re Still Friends ;-)

My peer review of your modules is complete and has been posted as individual feedback on Blackboard.

You do NOT have to make all the changes I suggested by the due date of the final project, nor should you try

Please remember that my goal here was to replicate the peer review process as much as possible. I thus responded like I respond to actual articles that come in for academic journals or conferences I am part of, which are the venues that all of you chose.

You might want to focus on 1-2 of my critiques and revise for those by the final due date for the project. DON’T TRY TO DO THEM ALL. It takes time to revise-and-resubmit, which is the response that all of you got.

Remember when I said that the write-up is the hardest part?

You have all conducted sound pilot studies. I would have told you by now if they weren’t sound, but writing about those studies is like a whole different skill set. You have to explain every in-and-out of your study and why you made the choices you did in a genre that you have never written in before.

Research articles and presentations are NOT the same as the papers you write in school, unfortunately. I think they should be, but this extra-curricular genre is so hard to write that I think a lot of faculty balk at trying to teach it. Really, we need a whole class in how to write these things. Not just one module in one class (or a few classes).

No one should even THINK about redoing your study

Don’t get discouraged by my peer reviews. No one gets write-ups right the first time, or probably even the 12th time, to be totally honest. And my reviews did focus on THE WRITING, not the research.

The way you describe your study, honestly, is more important than the way you conduct it. A lot of people would shoot me for saying that, but even if you do the best study in the world, if you never publish on it, no one will ever know about it, thus proving my point.

But you need to do an “actual” study before publishing, anyway

Please also remember that these are pilots, meaning that you CAN’T PUBLISH ON THEM IN ACADEMIC VENUES, YET. Some venues allow for the publication of pilot studies, but most don’t. If you’re not sure, write your venue’s editor and ask if they publish pilot studies.

The big hurdle is IRB: most academic publications require that research be conducted in consultation with an IRB.

That being said, IRB is just a submission away: http://www.ecu.edu/irb/. Well, the first thing you need to do is register for ePirate: http://bit.ly/1iCB7Cw. Here are directions for doing so if you’re unfamiliar with this system: http://myweb.ecu.edu/ohri/registration-ecuusers.mp4. When you do register, be sure you mark “I am a study coordinator or staff member of a research study team” as one of the roles you’re willing to fulfill.

Then you’ll need to complete your IRB training by going to http://www.citiprogram.org, creating an account, logging in, and completing the “Behavioral and social sciences investigators and key personnel basic course.”

Then, after all that, you can submit your study by going to ePirate: http://bit.ly/1s0pDwl. The just three short weeks later, voila! Yeah, it’s a lengthy, bureaucratic process.

Once you get IRB, you should collect new data

I often do pilots before conducting the “real” study, because they teach me a lot about that real study, which is often larger in scope. The real study is also an opportunity to get better data. What information did you need from participants that you failed to get? What are new kinds of participants you might recruit? Do you need to refine your research questions?

About the criteria I used for the peer reviews

The criteria I used are pretty standard for any academic venue. Let me explain a bit more about each criterion.

Significance/purpose: This is an assessment of what kind of contribution the research is for its specific venue. Is it new? Timely? Compelling? Does it build on existing scholarship, but contribute novel findings?

Methodology: This is an assessment of how sound the framework for your study is. Do the research questions match up with the theory, methods, and findings? Is there a clear connection between all these elements?

Findings: Are the findings articulated in sufficient detail that they are understandable, even to a well-educated non-specialist? Are they a direct response to the research questions? Are they significant? Is the “so what” of your findings clear?

Style: What about the language used in the study? Is it clear? Does it make as little use of jargon as possible? Are there concepts or passages that could be better explained?

Teacher Response to Module #4 and Homework #5: The Difference between a Research Report and a Research Article

Grades on Blackboard

Reporting vs. getting published

You’ve all created successful pilot studies and written up some very interesting findings. Unfortunately, these findings will never see the light of day if you don’t work 2x as hard to make them intelligible to a particular venue. Researchers read research articles and listen to research presentations. They don’t go beating down the doors of other researchers, outside of journals and conferences or meetups.

So, you’ve got to go to where your audience is, and you’ve got to learn how to sound like them. It’s as simple as that.

No one has original ideas

As some of you noticed, my recent article in Technical Communication was a direct imitation of the piece “Composing Across Multiple Media” that I assigned for Module #3. This is so taboo in the humanities that I hesitate even to point this out on our course website. It is the norm in the social sciences though, where methods and methodologies are considered communal property.

Why did I imitate Ranker so closely? It was an effective research design that worked for my study. Rather than reinvent the wheel, I took someone else’s wheel, measured it against my data, and said: hey, that fits pretty good!

Where a lot of training in the humanities gets it wrong is that we try to convince you to be “original” thinkers. This is like researching and writing in a vacuum, though. You’re going to create an “original” thought? Meaning something that’s never been thought before by anyone in the history of mankind?

More importantly: research doesn’t work that way. Research is a conversation. If you’re not willing to imitate, to take others’ ideas and test them out, then you’re talking to yourself.

#getsoffsoapbox

All this is to say: Try not to get upset when I respond to your drafts that are due on the 3rd

All this is building up to: I’m essentially going to give you a peer review of your drafts that you hand in to the website on the 4th. This means that I’ll be evaluating you based on the soundness of your research design and your write-up of that research design.

This will not be pretty. My goal will be to treat you as editors will treat you when you try to publish your first research article or present at your first major conference (WHICH ALL OF YOU ARE GOING TO DO, RIGHT?!). That is to say: I’ll be trying to find holes in your argument, gaps in your literature review, flaws in the way you’re describing your methodology, etc.

This is all part of doing research. Accepting peer critique and trying to improve your writing on your research is a necessary part of the process of becoming a researcher.

That being said: I will also include a “what I expect by the 11th” list, because you won’t be able to respond to all my feedback in just one week. The last time I got a revise and resubmit, it took me about a month to respond to all the changes, and being a researcher is essentially my full-time job.

Even though research is hard, writing about research is at least 2x as hard, in my personal experience. I’ve also been a serious researcher for about 10 years now, and I’m just now starting to get the hang of it. I don’t know if you ever completely get the hang of it though, honestly; you just learn how to roll with the punches better.

So, roll with the punches I send your way on the fourth and try to learn from my feedback. You all have done research that is way too good for it to never see the light of day!

When someone else does your research for you

A lot of researchers (including a past version of me) get really upset when someone else beats them to the punch with a project, but the more intermediate researcher I am today revels in it. If someone else has produced useful findings, why not use that to do a follow-up study to test their findings? There are always new opportunities for research, especially in fields related to writing or communication, which is constantly evolving.

So, here is a report based on a survey of 247 freelance writers on the state of freelancing today: The-State-of-Freelance-Writing-2014.pdf

It’s by http://www.copypress.com/, which is one of the good guys, at least from my initial research into this area.

Teacher Response to Homework #4

Grades on Blackboard.

Ethics Schmethics

When I talk to people in industry about the ways academics do research, they look at me as though I’ve just said “I love smallpox!” Industry researchers are all about solving problems and they don’t see why anyone would go through all the trouble to develop scripts, file IRB, etc.

Then I ask them what they’re doing and they say something like: “I’ve spent the last 9 months interviewing corporate executives about their workflow to help improve it.”

Me: “Did you reveal their identity to the public?”

Them: “No.”

Me: “Why not?”

Them: “They’re my client. That would be highly unethical.”

Me: “Exactly.”

My point is that everyone is invested in some form of research ethics, whether they realize it or not. Unless you’re a tabloid journalist, when you interview someone, you don’t want to defame them. Unless you’re a used car salesmen, when you survey a bunch of post customers, you don’t use that information to exploit them.

It is arguable that academics do things to extremes by overly bureaucratizing the process and requiring oversight on top of oversight on top of oversight. ECU is the second institution I’ve been at that has a “Committee on Committees,” whose job is make decisions about who’s making decisions.

What we do get right, though (at least most of the time, I hope), is research ethics, which should always protect participants, first, second, third, and forty-third. So, whenever you do a research project for the rest of your life, as yourself: is it in the best interests of your participants? Is there a potential to cause them harm? How can you mitigate that potential?

My personal assessment of the telecommuting opportunities mentioned before

Jessie mentioned some freelance opportunities in an earlier post, and mostly as a kind of research experiment, but also because there were some consulting and contract jobs on Flex Jobs that I thought I’d try to grab in my “spare time,” I signed up for Flex Jobs and a couple others. Below is my assessment of each one I tried. Other people might have had better experiences. Please feel free to share if you have.

Flex Jobs

  • Most of the jobs I found were linked to from elsewhere on the Internet through free sites like Monster and Simply Hired. Out of the dozens of jobs I examined, only two wanted applications directly through Flex Jobs.
  • Most of the jobs I found said “telecommuting,” but then when I clicked to go to the original posting, it turned out they wanted someone in a particular area.
  • The vast majority of jobs appeared extremely low paid. On a whim, for example, I investigated a “content writing” job through a firm that seemed legit, and through no effort on my part they “approved me” for this job. Turns out it was editing random entries online in sites like Tech How for $25 per entry. I estimated what that would be hourly for the work required for each entry and it was basically minimum wage or less (it’s not an hourly rate, so they can charge whatever they want–it’s not “their fault” if you end up making pennies on the dollar because their offering takes you hours to complete it).
  • They also offer “skill tests” to make you stand out to employers, but these skill tests were VERY difficult, even in areas in which I consider myself an expert. I flunked my first one, for instance, and got a B in the “technical writing test,” even though I have been doing that work for years.
  • After my preliminary examination, I asked for a full refund (which I did receive).

Scripted

  • At first, Scripted appeared to be a much better model. It was free to sign up for, and after a particularly fascist grammar test (which I also failed the first time), I was “approved” to write 500 word articles that appeared in my dashboard for close to $60 per article. I estimated I could do an article in 2-3 hours, meaning that I’d be earning $20-30 per hour. Not bad.
  • Then, however, I realized that I also had to apply for “industries” in order to land those articles. These industries are governed by anonymous “community moderators” who accept or decline you based on a writing sample which they judge for voice, clarity, flow, and reader engagement.
  • You actually have to submit two writing samples, one you’ve published somewhere and one generated in response to a prompt. After being told that my writing sample was “too short” (but not which one was too short), I lengthened it and then was quickly declined for the “industry” I applied to, which was business, a field I have numerous peer-reviewed publications in. Below is the “feedback” I received. There was no option to revise or resubmit.

Regarding the consistency of your voice: I know what voice this piece needed, but the writer didn’t execute it perfectly.

Regarding the clarity of your writing: I see what the writer was trying to express, but I’m still a little confused.

Regarding the flow of your prose: The thoughts linked together, with one or two notable exceptions.

Regarding the extent to which you kept the reader engaged: The subject was interesting and informative, but my focus still wandered.

oDesk/Elance

  • So, near as I can tell, both these sites are owned by the same company (oDesk Corporation). I signed up for oDesk, on a whim, which was also free and seemed built on a similar model to Flex Jobs with notable exceptions: the user interface was much more intuitive, it pushed jobs to me instead of me having to do numerous, time-intensive searches, and the first “skills test” I had to take was actually a very helpful tutorial in how to use their system to get work.
  • Results here are inconclusive because I haven’t applied for anything yet, but I will let you know my experiences, because there are some freelance opportunities on there I might apply for.

Suggestions

The best way to find freelance work, in my personal experience, is to get out there in your backyard and find organizations that need work done. Now, full disclosure: I have never made a lot of money as a freelancer. There are faculty who have done so at ECU and elsewhere, so if you want to know how to make money at freelancing, you should probably talk to those folks ;-).

Having talked to a few of them, myself, they all seem to all say similar things:

  • You have to develop recognizable expertise within a particular field (meaning, recognizable beyond academia).
  • You have to find actual people out there willing to pay you money for that expertise.
  • Getting your first freelancing gig is always the hardest, but once you develop a portfolio of work and some testimonials from past clients, you can leverage that into a steady stream of work.

I’m considering turning these anecdotal experiences into a full-blown research project about freelancing. If I do that, you’ll definitely hear more about that.

Again, if anyone else has had different experiences with freelancing, via the above sites or elsewhere, please let us know!

Teacher Response to Module #3

Grades + individual feedback on Blackboard.

Again, I’ll save you the trouble

My comments to everyone were pretty much “sounds good; now go collect some data!”

As I mention in the next Module, the most important things to consider during data collection are the following:

  • Doing what you said in your plan you were going to do. Even though this is a pilot, it’s important you try out the research design you’ve constructed.
  • Protecting your participants from harm. This means ensuring confidentiality, minimizing risk, and making sure they understand why you’re collecting information on them.
  • Not collecting too much data. Remember that this is a pilot. Your goal here is really to test out your research design, not to collect a ton of information that you’d then have to recollect to turn this into an actual study.

Have fun in the field! Let me know if you step in any bear traps!

Teacher Response to Module #2

Grades on Blackboard. The responses were pro forma…. see below.

I see research people*

You may have by now uncovered my secret plot behind this class, and that is to get you all so excited about empirical research that you can’t wait to do it for the rest of your life.

One of my firm life philosophies is that if everyone would do everything more empirically, meaning based on actual data they had collected themselves, then we could solve all the world’s problems. The opposite of empiricism, ideology based on opinion or doxa, is what creates and maintains many of the world’s problems.

I am overjoyed by the studies I am seeing and I can’t wait to see research instruments and eventually data. I hope you folks are as excited as I am by your studies, but I’m not sure that’s completely possible ;-).

Great work.

*In the tradition of “I see dead people meme

Teacher Response to Homework #4: Aaaaaah!!

Grades on Blackboard.

What is all this jargon?!

If you’re anything like me, at this point in the process of this class you may feel awash in a see of jargon. Attitudinal scales. Variables. Research instruments?!

It may sound like a different language. And guess what: it is. Those of us who have been researchers for a number of years have absorbed this very alien discourse and are “fluent” in it, so to speak. But it IS foreign, and it CAN be alienating.

Which is why research can be done poorly

Several of you have commented in your homework assignments that you’re starting to see flaws in published research based on what you’re learning in this class. Remember what our guest speaker said: it’s easy to do research wrong.

As you work to understand the concepts central to research design, please remember that people disagree on these concepts all the time. What makes for a valid study? How can a study be truly replicable? How extensive does a research instrument need to be?

Questions such as these have answers, but they are not universally-agreed upon answers.

Now you’re becoming part of the conversation

To find your way in this mess of terms and complex processes, please remember that the best way to learn any language is to jump in and use it. As you become part of this conversation, remember that you don’t have to agree with everyone else’s interpretation. That’s what fuels research, actually: collegial disagreement. So, if you think a term is poorly defined in our textbook, write it down for later study. If you’re not so sure attitudinal scales are valid means of analysis, look up critiques of them.

Be a researcher of research design, first and foremost. That’s the best way to become a good researcher.

Teacher Response to Module #1

Grades on Blackboard. Please also check for your individualized feedback there.

Some overall stuff to think about is below.

As Dr. Grabill said, research is hard to do well, and easy to do poorly

Almost everyone got their designs tweaked in one direction or another. And that’s okay. The best way to learn how to do research is to do it with someone with a bit more experience looking over your shoulder.

Your studies are evolving, and that’s okay, too

Many of you submitted documents that show that your thinking regarding what you are studying has shifted. Try not to get frustrated with that, but instead embrace changing your mind. Mostly what I am doing in this class, is pushing against your thinking to help you refine in and create sound studies. This process is, by necessity, messy and iterative.

Teacher Response to Homework #2

Grades on Blackboard, as per the norm.

Below is some stuff to think about.

Variables often emerge after a pilot

The reason we’re doing pilots in this class, as opposed to full-blown studies, is that pilots, or pilot-like activities, are often essential to defining variables. It’s often hard, when designing a study plan, to figure out what exactly you’re measuring, and there are typically a lot of different variables floating around in your study that you could use. The choice often comes down to what you are most interested in, and what you want to zero in on.

Best practices are important, too

That being said, what you decide to zero in on, and how you decide to collect data, should come from best practices within your field. As you are learning, research design is a complex activity, so the best way to do it well is to look to the successful research designs of others. Don’t  try to invent in a vacuum, because if you do, chances are your design will not be as good as it could have been.

Here are just a few things to watch out for:

  • The specific way you deploy a particular method. What are best practices for deploying this particular method and how are you responding to those best practices?
  • The variables you end up choosing, as compared to the variables of other researchers. If you’re studying variables that have never been studied before, chances are you are actually studying concepts, and have yet to define actual variables. The best way to avoid this is to mine other studies for variables.
  • The theory you’re using and how it is used within your specific field. Theories are some of the most politicized elements of research design. They come in and out of fashion, are deployed differently in different fields, and are often deployed poorly. To avoid this, look to researchers studying similar things and what theories they’re using. Avoid taking a theory from a completely different context and jamming it into your study because it’s something you’re familiar with.

Looking forward to seeing the study plans!