Online courses, automating education, and digitalizing degrees

Online teaching is here to stay and I don’t think anyone knows how it will go.

Some schools will take risks, some will pull ahead, some will stay the same, some will lose. Teaching is moving to the digital domain, so we will see more digital effects - and our institutions are not designed around it. 

Consider this phenomena around the attendances of Introduction to Artificial Intelligence (CS 188) over the summer at UC Berkeley. We are offering a class with broad discussion times to encourage all students around the world to participate and recording all lecture and discussion material.

• Enrollment: ~190

• Lecture attendance by day: 120, 95, 87, 75, 56, …

• Discussion attendance mirrors this number in sum (low engagement and the running theory is that the same people who go to lecture go to discussion, not half and half). 

What are 60% of students doing? Are they just watching recordings? For reference, when I lectured this class last spring to 800 people enrolled (on Zoom), 100 watched live and the recordings rarely hit 250 views. 450/800 students have seen neither before the final. This was before our student body was spread across the globe, and as a staff we have 0 insight into this, which is quite the predicament for an education institution. This leads into major questions:

• What are the majority of the students doing? 

• Are students choosing to make this an online course, do they have conflicts of time commitments or conflicts of timezones?

• Should universities accomodate every student’s time preferences in a global course?

I want to engage on three main themes:

1. The behavior in online courses,

2. How these courses are leaning towards automating education,

3. and the implications of digital degrees.

---

Behavior and Online Courses:

Online courses are a wild-west for students. It’s the merging of trends of reduced community in courses with increased digital engagement among individual students. Students learn and feel valued by having opportunities to engage. A blog post summarizing the trends in research on engagement is to the point (check it out for the studies it references):

Research has historically indicated strong correlations between student engagement (typically defined as attention to the area of focus, active participation in learning, and time on task) and student achievement. These correlations remain strong for all levels of instruction, across all subject areas, and for varying instructional activities.

That is the problem we are unintentionally contending with. Current Universities are structured for in-person engagement (giant lecture halls aside) but this holds no weight in the transition to online classes. See more about the links between engagement and learning in terms of in-lecture responses and of drop out rates.

(Online courses can probably learn from live-streamers, such as 3blue1brown).

Zoom-room Atmosphere

There is no question that students have less of a barrier to asking questions through a chat box than in person, but I think it leaves a disconnect. There is feedback loop between the subtle emotional cues of students and the instructor. You cannot replicate the feeling of 100 confused faces looking up at you after you sped through one too many lecture slides (speaking from experience, it’s daunting). This mismatch has consequences.

Consider this quote from a Data 100 instructor (thanks Vikram) in the spring:

“ I think TAs need more help create that sort of [actively engaged] atmosphere. e.g., in a classroom it’s really easy for me to go up to the board and scribble out an example, but I can’t do that if we’re on Zoom (even with an iPad, not super easy unless you have the Apple Pencil).”

Where does this help come from? Does the help come from the chat box? When thinking about in person questions vs zoom chat questions, we feel:

“it’s really easy to ask a question and to get students to 👍🏽or 👎🏽it on Zoom — but I’m not sure those are actually helping foster a ton of engagement”

This goes back to where I started. Are the students playing a passive game on Zoom while we lecture to them, or is this still a useful form of teaching.

Zoom Freeloading, Students Leaving, and Balance

Students come and go from Zoom rooms as if there is no social barrier, and that’s because there isn’t - there’s a digital button.

A question I am wondering is: What is the sweet spot where people don’t leave / will engage / don’t free float in a discussion section? Ie, what is the optimal number of students in an online discussion considering the leaving mechanisms (too empty or too full) and the freeloader effect. My hypothesis is 8-12 students, which is much less then that of in person discussions (amplified effects that were already in place because its easy to leave and easy to not contribute).

The problem is, this does not scale to bigger classes - there already are not enough instructors! So, should we still host discussions is students are going to watch recordings?

Student Synchronicity vs Staff Coverage

Content repetition in a section is of less value if everything is recorded, online, or in a published notebook. Should classes be designed to be fully synchronous (students do material with eachother) or to cover every student’s preferences.

When setting up a course, staff generally wants each student to be able to attend every option. Normally, the only constrain is other courses, now the big constraint is global timezones. When setting up discussion times at the beginning of the semester, the question is if coverage is an access problem or a staffing problem.

• Access problem: do individual students need to have times when they can attend section? How much pressure on access should they have.

• Staffing problem: due the new distribution, do we need more staff or more distributed staff?

A small story about designing times for CS 188 - introduction to artificial intelligence - this summer.

We used Piazza to poll timezones (there are no great options, but this is not the one to use). Unsurprisingly, this showed a focus around Pacific Time, but there were students that considered every timezone around the world a working hour. The problem with the Piazza tool is that it does not record identities per hour slot - therefore you cannot see which students have coverage based on buckets of section times (the ultimate goal is to increase this percentage). I don’t think anyone at a University will want to make discussion times exactly proportional to preferences (because there are so few instructors that rounding will leave people out), but the task of guaranteeing every student to have a slot is tough.

Or, consider the approach of my friend Professor David Delchamps at Cornell University: he records his lectures alone and releases them so the students that can make the lecture time are not at an advantage, and the lecture slot is now office hours. These design decisions are going into place with no overhead from departments and no studies to reference.

Minimizing Status-based Advantage

The question everyone needs to be asking while we get sparse signals back from students, instructors, and professors is: how is this effecting underrepresented groups that don’t have a microphone to express their concerns. Two mechanisms come to mind.

1. Lack of accessible technology for digital input (Three New York Times Pieces: desire to be at school, online learning’s effect on the mind, on public vs. private schools).

2. Methods for interacting in course materials, e.g. typing questions in chat rather than raising ones hand in a crowded hall (this could be an equalizer, but should be confirmed).

Additional Comments

A) How does credibility of staff remain after a miss up with a deadline etc when there’s no in person network? Are students more or less forgiving online? How does this evolve semester-to-semester?

B) Hallway effect of schools. What benefits do random interactions have and who do they benefit? Should there be discord channels to chat in? Can the university facilitate this and do network effects decay on a per-class basis (suggesting some permanence to the bonds), faster (suggesting seeing people always is important), or slower (suggesting there are more complicated effects at play)?

---

Automating Education

Industry analysts have pinned education as the most difficult industry to automate [source, source]. I distilled this: we can automate learning, but we cannot automate education. Education is a system, learning is an about a subject.

Who benefits from massive online open courses (MOOC) and who benefits from structured, incremental education? Take a second to formulate your own prior.

Those who want to learn about a specific subject matter benefit from MOOCs of today (think someone learning computer science skills), but I haven’t heard of one person who gained their scientific ability and critical thinking from an online course - those are from structured (breadth) learning that allows for freedom (depth in arguments). People take existing abilities and combine them with rapid, available online learning to pivot careers or make products.

Robot Tutors

Some people think robot tutors can solve this (reading on robots teaching), but I think this is very, very far from the truth. Robots can provide information, but they cannot process the complex interaction of teaching. Online courses are pretty much robots trying to teach people on confined tracks - this doesn’t scale well. The language processing abilities are limited in AI now, and there are no content creation AI’s, so I really doubt intricate automatic learning systems will be comparable (simple ones are still valuable yes).

Easy for robots:

• Grading assignments and generating permutations of existing questions .

• Time-adjusting material based on performance: keep performance scores in a range, and accelerate if ahead and slow down if behind.

• Individualization: branching of a tree as educational path.

Hard for robots:

• Emotional capacity and individualization (personal needs). This will be a problem that accrues and accumulates - backwards pointer to why we want to democratize automation.

• Underrepresented groups and access problems (e.g. training set will be for Type A, hard working students, not students with learning disabilities or non-conventionally structuring of arguments). 

As Universities become more like online courses, the education system will resemble more of just learning things rather than engendering intelligent process. This is a slippery slope I am afraid of.

Some more related articles: the education/culture tradeoff in Flanders, a culture in education study on African American students, a Washington Post article on the point of education beyond learning.

---

Digital Degrees

When writing this I got an email from UC Berkeley that the number of COVID19 cases on campus jumped from a cumulative total of 23 prior to July 2020 to 47 new cases in the last week (traced to Greek Life parties). I see the public school systems and Universities in large urban centers to be hard pressed to open in the next 12 months.

A divide among giants

This is a brief section, because it’s mostly just recent events. Many schools are announcing their plans for the fall. Two big examples are Harvard going online and Cornell opening. Cornell’s surprise announcement was based on a survey showing most students would still want to come to campus if online (that’s a big value-statement from students in terms of what campus means to them). Cornell then will use this physical presence to monitor cases and provide medical help, rather than students being on campus and around without official opening.

The ability to open comes from an isolated location. Schools in cities like Harvard will have a hard time opening (even though Columbia has said otherwise, I think it is a risk).

Tuition prices, valid complaints, and education brands

You know there’s tons of jokes around about the value of educations when they go online, and there’s a couple reasons it’s so hard - the difficulty is disambiguating the different values that different individuals receive. Some students benefit entirely from the brand (and connections - stereotypical student-athlete at Harvard), which would warrant maintaining tuition (or raising in some school’s cases), but many students do actually benefit from an education (with lower than $50k/year value).

This gap is impossible to quantify, but individuals will make it clear when they do not enroll in some schools, when some schools close, and when some schools are forced to innovated (and maybe undercut on tuition and aim for transfers to boost income).

A digital university?

I’ll leave this essay with a thought: will any mid-tier universities opt for increased digital enrollment at the cost of reputation? The process is: dramatically lower tuition (~10%) and make transfer acceptances high and of low bureaucratic burden. Could this steal students from other schools when the class material is all digital and easy to reproduce?

---

What’s new with me

I found this quote from an NLP researcher very interesting in the context of neural network theory - and I am definitely going to dig deeper (subscribe for the survey).

Most results [in neural net theory] are to the effect of: as the number of parameters goes to infinity, local optima are as good as global optima with high probability. There’s good progress in neural net theory, based on thinking about the limit of infinitely-wide nets. A lot of this work uses the term “neural tangent kernel.”

I am reading (newsletters/blogs):

• James Clear’s “Habit Scorecard” - I wish I knew how long lockdown would’ve been and started here.

• A visualization of state-vs-state supreme court cases (and who is winning the most).

• Investment in AI hardware has driven cost down at a crazy exponential rate (bodes well for accessibility to individuals for projects and research).

Books:

• Human Compatible: Artificial Intelligence and the Problem of Control - Stuart Russell: I’m intrigued by his formulation of a new reinforcement learning paradigm that does not leave a risk of humans being outfoxed by AI - the idea is to make the robots very unsure of themselves and have imperfect models for human preference. I recommend the book.

I am listening to / watching:

• Hamilton on Disney Plus. Totally awesome.

---

Find more about the author here. Forwarded this? Subscribe here.