WEBVTT 1 00:00:05.930 --> 00:00:10.930 And, and the presentation is on large language models and 2 00:00:11.560 --> 00:00:15.280 neurological mimicry for PhD student advising. 3 00:00:15.280 --> 00:00:18.930 And with that, I'll hand it over to Justin. 4 00:00:18.930 --> 00:00:19.530 Great. 5 00:00:19.530 --> 00:00:22.490 Thank you so much for that introduction and thanks for the 6 00:00:22.490 --> 00:00:27.490 opportunity to present our research or rather our design as 7 00:00:28.220 --> 00:00:30.170 it is progressing. 8 00:00:30.170 --> 00:00:35.170 Hopefully you can all see my screen just fine and I'll, 9 00:00:36.640 --> 00:00:40.830 I'll hope someone will unmute and tell me if you can't. 10 00:00:40.830 --> 00:00:45.830 I sent the link in the chat and so on slide two, 11 00:00:46.380 --> 00:00:50.480 you have a chance you're invited to sort of log your level 12 00:00:50.480 --> 00:00:54.680 of fam familiarity and your level of interest with these 13 00:00:54.680 --> 00:00:58.410 seven facets that will be touched on in the talk. 14 00:00:58.410 --> 00:01:01.110 So if you're able to do that, 15 00:01:01.110 --> 00:01:03.220 I mean if you're on an iPhone that's gonna be tricky, 16 00:01:03.220 --> 00:01:08.220 but if you're in front of a larger screen, just grab a dot. 17 00:01:09.400 --> 00:01:13.140 And maybe you're not very familiar, 18 00:01:13.140 --> 00:01:15.790 but you are very interested. 19 00:01:15.790 --> 00:01:20.790 A high level on this side and modest on the outsides. 20 00:01:22.800 --> 00:01:27.800 I'll give you a minute or so to log your interest. 21 00:02:03.210 --> 00:02:05.050 Don't worry, you don't need to be polite. 22 00:02:05.050 --> 00:02:07.090 If you just have a modest level of interest for something 23 00:02:07.090 --> 00:02:07.530 that's fine. 24 00:02:07.530 --> 00:02:09.850 Looks like everyone's starting with their level of 25 00:02:09.850 --> 00:02:12.250 familiarity and then they're moving over to the affective 26 00:02:12.250 --> 00:02:13.410 side. Second. 27 00:02:13.410 --> 00:02:14.493 Totally good. 28 00:03:03.430 --> 00:03:06.030 Kudos to whoever put a line. 29 00:03:06.030 --> 00:03:06.270 Excellent. 30 00:03:06.270 --> 00:03:08.880 You could draw a line through and snake it through. 31 00:03:08.880 --> 00:03:12.380 It represents all four of those quadrants. 32 00:03:17.510 --> 00:03:19.843 We'll do another 30 seconds. 33 00:04:02.060 --> 00:04:03.690 Okay, great. 34 00:04:03.690 --> 00:04:04.370 I'm gonna, 35 00:04:04.370 --> 00:04:09.060 I'm gonna believe that most everyone has sort of logged in. 36 00:04:09.060 --> 00:04:13.300 Looks like there's high level of interest in almost all the 37 00:04:13.300 --> 00:04:18.300 areas and maybe a medium level to a modest level of 38 00:04:18.980 --> 00:04:22.630 knowledge. Y'all are probably just being modest. 39 00:04:22.630 --> 00:04:27.380 So let's get started on slide three. Oh, by the way, 40 00:04:28.260 --> 00:04:33.260 slide 71, spoiler alert, has the different iterative demos. 41 00:04:34.430 --> 00:04:38.060 So if I'm droning on and you want to try a demo, 42 00:04:38.060 --> 00:04:41.760 you can go right there and try them. I'm not trying to, 43 00:04:41.760 --> 00:04:44.180 you know, shift you from what I'm, 44 00:04:44.180 --> 00:04:45.320 what we're gonna talk about, 45 00:04:45.320 --> 00:04:49.260 but just know that that's where we're headed. 46 00:04:49.260 --> 00:04:52.170 Okay, so here's an agenda. 47 00:04:52.170 --> 00:04:57.170 I'm gonna go fairly quickly through these slides and feel 48 00:04:57.350 --> 00:05:02.350 free to unmute and ask questions now or at the end, or both. 49 00:05:02.750 --> 00:05:05.190 So here's our timeline. We started, 50 00:05:05.190 --> 00:05:10.190 this is a fairly new line of research started just last 51 00:05:10.220 --> 00:05:15.220 August. And in terms of design started in December of 22. 52 00:05:15.470 --> 00:05:19.840 Right now we're in the prototype development phase. 53 00:05:19.840 --> 00:05:24.840 So why PhD advising? I'm gonna turn it over to Faru, 54 00:05:24.870 --> 00:05:26.453 take it away. Faru. 55 00:05:30.890 --> 00:05:32.770 Do I need to share my screen? 56 00:05:32.770 --> 00:05:35.440 No, you just tell me when to move this slide ahead. 57 00:05:35.440 --> 00:05:38.920 Would you please this Number seven. 58 00:05:38.920 --> 00:05:39.910 Great. 59 00:05:39.910 --> 00:05:41.130 Okay, 60 00:05:41.130 --> 00:05:46.130 so advisor advise the relationship is crucial for the 61 00:05:47.920 --> 00:05:49.460 success of graduate students, 62 00:05:49.460 --> 00:05:53.320 but increasing number of graduate students and limited 63 00:05:53.320 --> 00:05:58.320 faculty positions have led to challenging advisor advisee 64 00:05:59.310 --> 00:06:04.310 ratios, potentially reducing access to advisors. 65 00:06:04.690 --> 00:06:05.960 So as a result, 66 00:06:05.960 --> 00:06:10.710 it is important to investigate doctoral level advising 67 00:06:10.710 --> 00:06:14.960 dynamics due to three main reasons. First, first of all, 68 00:06:14.960 --> 00:06:19.880 the importance of successful advising experience experiences 69 00:06:19.880 --> 00:06:22.060 for students development, 70 00:06:22.060 --> 00:06:27.060 the primary role of doctoral advising in professional and 71 00:06:27.520 --> 00:06:29.420 disciplinary development, 72 00:06:29.420 --> 00:06:34.420 and the impact of advising and student attrition and their 73 00:06:34.480 --> 00:06:35.980 degree completion. 74 00:06:38.050 --> 00:06:39.550 Next slide please. 75 00:06:40.780 --> 00:06:45.780 So there are some challenges with traditional advising 76 00:06:45.930 --> 00:06:47.640 process. First of all, 77 00:06:47.640 --> 00:06:52.330 advisors spend most of their time dealing with recurring 78 00:06:52.330 --> 00:06:56.700 questions related to basic issues such as courses, schedule, 79 00:06:56.700 --> 00:07:00.770 or mainly administrative related questions. 80 00:07:02.880 --> 00:07:06.860 And high advisor advise loads. 81 00:07:06.860 --> 00:07:11.860 And it is also very error prone because sometimes advisors 82 00:07:12.880 --> 00:07:17.880 may lack awareness of the changes regarding the degree 83 00:07:18.360 --> 00:07:22.460 requirement, study plans and other regulations. 84 00:07:22.460 --> 00:07:27.120 And also there there is a long awaiting period at advisory 85 00:07:27.120 --> 00:07:28.440 office A. 86 00:07:28.440 --> 00:07:33.100 And also the last one is adequate meetings and still it's 87 00:07:33.100 --> 00:07:36.720 due to the advisors spending a lot of time, 88 00:07:37.560 --> 00:07:40.410 most of their time doing administrative. 89 00:07:40.410 --> 00:07:45.410 So they do not have time for the, like an actual meaningful, 90 00:07:46.550 --> 00:07:51.440 more meaningful advise advisor interaction. So. 91 00:07:53.180 --> 00:07:54.013 Great. 92 00:07:56.570 --> 00:07:59.153 Okay, so large language models, 93 00:08:00.220 --> 00:08:05.220 you have likely heard about them and the Google brain paper 94 00:08:06.660 --> 00:08:10.620 of 2017 attention is all you'll need is usually pointed to 95 00:08:10.620 --> 00:08:13.260 as one of the catalysts for where we are currently with 96 00:08:13.260 --> 00:08:15.510 large language models. 97 00:08:15.510 --> 00:08:19.260 Here's an example of the attention mechanism. 98 00:08:20.400 --> 00:08:25.350 Transformer architectures were the thing that was pinpointed 99 00:08:25.350 --> 00:08:30.350 as as key and then having multiple attention mechanisms. 100 00:08:31.150 --> 00:08:35.260 Here's one showing just just for the word making. 101 00:08:35.260 --> 00:08:36.550 It's sort of a look ahead. 102 00:08:36.550 --> 00:08:38.430 So it's looking ahead a few words, 103 00:08:38.430 --> 00:08:41.790 but it's also looking ahead and making predictions far along 104 00:08:41.790 --> 00:08:43.040 down the chain. 105 00:08:44.060 --> 00:08:47.070 You'll notice that in terms of these models, 106 00:08:47.070 --> 00:08:50.830 they've been growing by about a hundred x per year in terms 107 00:08:50.830 --> 00:08:52.330 of parameter size. 108 00:08:53.880 --> 00:08:57.720 And this means that they take a lot of energy to run, 109 00:08:57.720 --> 00:09:00.800 to compute to, to get going. 110 00:09:00.800 --> 00:09:03.960 You, this probably looks familiar, 111 00:09:03.960 --> 00:09:06.550 this is the highest profile, large language model, 112 00:09:06.550 --> 00:09:08.590 open AI chat, G P T, 113 00:09:08.590 --> 00:09:13.590 some people call it chatty powered by either 3.5 or G 114 00:09:15.490 --> 00:09:19.820 P T four version of their large language model. 115 00:09:19.820 --> 00:09:24.820 So what is a large language model in general? And again, 116 00:09:24.920 --> 00:09:28.970 down below I've got a little more text if you want more than 117 00:09:28.970 --> 00:09:30.490 what I'm just saying. 118 00:09:30.490 --> 00:09:31.050 But in general, 119 00:09:31.050 --> 00:09:34.650 large language model is a neural net trained on lots of 120 00:09:34.650 --> 00:09:37.010 unlabeled text. And when I say lots, 121 00:09:37.010 --> 00:09:41.730 I mean like common crawl is snapshots of the entire 122 00:09:41.730 --> 00:09:44.930 internet. So everything on the Mayo Clinic's website, 123 00:09:44.930 --> 00:09:49.290 everything on stack overflow, Wikipedia, thousands of books. 124 00:09:49.290 --> 00:09:49.920 Interestingly, 125 00:09:49.920 --> 00:09:53.670 a lot of unpublished books and some types of books are 126 00:09:53.670 --> 00:09:55.003 overrepresented. 127 00:09:56.530 --> 00:10:01.530 But let me continue. So once this language model is built, 128 00:10:01.540 --> 00:10:05.457 it can receive an input and generate an output. 129 00:10:06.300 --> 00:10:10.620 And that's the whole point of chat. G P T, 130 00:10:10.620 --> 00:10:15.570 you ask it a question, it synthesizes, it gives an output. 131 00:10:15.570 --> 00:10:20.420 One of the things that they've done maybe more recently is 132 00:10:20.420 --> 00:10:22.870 included R L H F, 133 00:10:22.870 --> 00:10:27.790 which is reinforcement learning from human feedback and it 134 00:10:27.790 --> 00:10:32.430 takes text input and it gives a bunch of different possible 135 00:10:32.430 --> 00:10:37.430 outputs and then a human sort of scores those and punishes 136 00:10:38.000 --> 00:10:41.910 or rewards the model based on, you know, 137 00:10:41.910 --> 00:10:45.710 what they think about the output that was given in order to 138 00:10:45.710 --> 00:10:49.430 get an even better output. Or at least that's the goal. 139 00:10:49.430 --> 00:10:51.910 So here's an example. 140 00:10:51.910 --> 00:10:55.980 If you haven't seen how ache G P T works and also an 141 00:10:55.980 --> 00:11:00.450 possible example of influence of R l HF on LLMs, 142 00:11:00.450 --> 00:11:05.450 here's a prompt about someone getting stung by a wasp. 143 00:11:05.690 --> 00:11:10.690 And then if we go to the G P T 3.5 response, 144 00:11:10.860 --> 00:11:13.030 this is what it responds, it responds more. 145 00:11:13.030 --> 00:11:17.090 But this is, this is the beginning of the response. 146 00:11:17.090 --> 00:11:21.580 This was in 22 December of 22, okay, 147 00:11:21.580 --> 00:11:26.180 then in March of 23, here's G P T four's response. 148 00:11:29.910 --> 00:11:32.820 So in general, G P T four works better, but you know, 149 00:11:32.820 --> 00:11:36.180 this response is more cautious, less specific. 150 00:11:36.180 --> 00:11:37.900 Maybe you feel good about this, 151 00:11:37.900 --> 00:11:42.540 maybe you feel disappointed about this. But the idea is, 152 00:11:42.540 --> 00:11:45.510 is that it's been trained to, 153 00:11:45.510 --> 00:11:50.510 to give a more appropriate response through that human 154 00:11:50.940 --> 00:11:52.410 feedback loop. 155 00:11:52.410 --> 00:11:56.980 Some folks prefer 3.5 because of this, 156 00:11:57.900 --> 00:12:01.580 obviously others don't. Okay, so in addition to this, 157 00:12:01.580 --> 00:12:04.440 when you give text input to a large language model, 158 00:12:04.440 --> 00:12:08.380 you can also output a numeric representation of that same 159 00:12:08.380 --> 00:12:10.480 text called an embedding. 160 00:12:11.500 --> 00:12:14.240 And we'll look at that in a sec. Okay, 161 00:12:14.240 --> 00:12:18.450 so project related data, some of the usual suspects, 162 00:12:18.450 --> 00:12:18.990 you know, 163 00:12:18.990 --> 00:12:22.710 if you're gonna have a bot that's gonna know things about a 164 00:12:22.710 --> 00:12:25.030 program, it's gonna need to know about the students, 165 00:12:25.030 --> 00:12:27.510 the faculty, the milestones, the course, et cetera, 166 00:12:27.510 --> 00:12:28.343 et cetera. 167 00:12:29.360 --> 00:12:31.990 So those things go in. 168 00:12:31.990 --> 00:12:36.990 One of you said you knew a lot about PhD advising and so 169 00:12:37.620 --> 00:12:40.250 many of these things will look really familiar, 170 00:12:40.250 --> 00:12:43.110 the different milestones that would go along with with 171 00:12:43.110 --> 00:12:45.630 getting a PhD. And for most PhD students, 172 00:12:45.630 --> 00:12:47.920 they only do these things once in their life. 173 00:12:47.920 --> 00:12:50.350 So they're not good at them. And being a PhD student, 174 00:12:50.350 --> 00:12:54.390 they're very nervous about being good at every at it because 175 00:12:54.390 --> 00:12:55.710 they feel like, you know, 176 00:12:55.710 --> 00:13:00.560 they have a high degree of competency in most other areas. 177 00:13:00.560 --> 00:13:04.260 And then here's an example of where each faculty member 178 00:13:04.260 --> 00:13:06.420 might have, you know, 179 00:13:06.420 --> 00:13:09.500 distinct opinions and preferences for how each of the 180 00:13:09.500 --> 00:13:12.510 milestones is carried out by their specific students. 181 00:13:12.510 --> 00:13:15.660 So it can change. So having the milestones is good, 182 00:13:15.660 --> 00:13:19.380 but not having faculty insight into how they navigate them 183 00:13:19.380 --> 00:13:22.940 or co navigate them with their student is also very 184 00:13:22.940 --> 00:13:23.773 important. 185 00:13:25.600 --> 00:13:30.490 Okay, so how do we access store and interact with this data? 186 00:13:30.490 --> 00:13:32.730 There are lots of different types of databases. 187 00:13:32.730 --> 00:13:37.250 Maybe the most used one is a sequel database followed by 188 00:13:37.250 --> 00:13:40.610 document graph and vector databases. 189 00:13:42.330 --> 00:13:47.120 Some of the different brands of these databases I list 190 00:13:47.120 --> 00:13:48.500 above. 191 00:13:48.500 --> 00:13:53.500 And for our project we use n E DB for telemetry data and Neo 192 00:13:55.000 --> 00:13:59.520 four J for graph content relationships and pine cone for 193 00:13:59.520 --> 00:14:01.103 content embeddings. 194 00:14:04.790 --> 00:14:07.940 So in terms of a graph database, 195 00:14:09.420 --> 00:14:11.460 it's sort of what it sounds like. 196 00:14:11.460 --> 00:14:16.460 It's a graph of the information with nodes being nouns and 197 00:14:18.160 --> 00:14:21.640 information and usually the relationships being the way they 198 00:14:21.640 --> 00:14:22.760 relate to each other. 199 00:14:22.760 --> 00:14:27.220 You'll note that that in this type of a graphing database, 200 00:14:27.220 --> 00:14:30.340 the relationships go in a single direction. 201 00:14:30.340 --> 00:14:32.960 You can have relationships that go in the other direction 202 00:14:32.960 --> 00:14:37.960 with some different name, identifying them if you like. 203 00:14:40.030 --> 00:14:45.000 Okay, so how do graph databases sort of, 204 00:14:45.000 --> 00:14:50.000 how do they line up with the brain if some of the advances 205 00:14:50.360 --> 00:14:52.920 that have been made with large language models have been 206 00:14:52.920 --> 00:14:55.760 based on this type of thinking, 207 00:14:55.760 --> 00:15:00.080 what should we know about graphing databases and how they 208 00:15:00.080 --> 00:15:01.120 match up? Well, 209 00:15:01.120 --> 00:15:04.200 they're network-based structures and you can kind of see 210 00:15:04.200 --> 00:15:08.760 that, that both of them inherently are based on networks. 211 00:15:08.760 --> 00:15:12.880 They're also decentralized and distributed in terms of how 212 00:15:12.880 --> 00:15:14.380 they store things. 213 00:15:16.430 --> 00:15:18.850 You might think of neurons and nodes, 214 00:15:18.850 --> 00:15:23.850 they sort of hang out together in terms of how they transmit 215 00:15:24.190 --> 00:15:26.190 and process information. 216 00:15:27.120 --> 00:15:31.760 And you might think about synapses and relationships, 217 00:15:31.760 --> 00:15:36.760 relationships within graph databases can have strength and 218 00:15:38.460 --> 00:15:40.930 you can have more than one with the same label. 219 00:15:40.930 --> 00:15:43.180 They can have as many labels, you know, 220 00:15:43.180 --> 00:15:48.130 or as many relationships going out from nodes as you like. 221 00:15:48.130 --> 00:15:50.760 There's also sort of, you know, 222 00:15:50.760 --> 00:15:54.800 in in the brain there's neuroplasticity in a graph database. 223 00:15:54.800 --> 00:15:55.600 You can update, 224 00:15:55.600 --> 00:15:58.080 you could update the strength of relationships, 225 00:15:58.080 --> 00:16:01.480 you can update which nodes exist in it, 226 00:16:01.480 --> 00:16:04.880 the connections between them, et cetera. 227 00:16:04.880 --> 00:16:07.210 This isn't to say that it's a perfect match. 228 00:16:07.210 --> 00:16:09.890 There are obvious differences including, you know, 229 00:16:09.890 --> 00:16:13.790 biological chemical or relational complexity. 230 00:16:13.790 --> 00:16:17.970 The, what is it? 231 00:16:17.970 --> 00:16:22.810 The brain has 86 billion neurons maybe and maybe winnows 232 00:16:22.810 --> 00:16:26.190 itself in adults down to 600 trillion synapses. 233 00:16:26.190 --> 00:16:29.330 The largest graph database is far from this, 234 00:16:29.330 --> 00:16:31.710 although catching up. 235 00:16:31.710 --> 00:16:34.990 Okay, so graph database for this project, 236 00:16:34.990 --> 00:16:37.110 what does it look like? 237 00:16:37.110 --> 00:16:42.110 This is the framework for PhD student advising. 238 00:16:43.730 --> 00:16:46.813 I'll just pause on this for a second. 239 00:16:52.880 --> 00:16:56.560 The first way we concentrated was thinking about students 240 00:16:56.560 --> 00:16:58.190 and topics. 241 00:16:58.190 --> 00:17:02.810 And so we started with students topics would be research 242 00:17:02.810 --> 00:17:05.150 topics, possible research topics. 243 00:17:05.150 --> 00:17:09.970 And we wanted everything to flow out from the student so 244 00:17:09.970 --> 00:17:11.850 that we could find the different paths. 245 00:17:11.850 --> 00:17:16.850 Because what we were thinking is, is that we want to give, 246 00:17:17.060 --> 00:17:20.170 we want to capture a path a student might take in terms of 247 00:17:20.170 --> 00:17:20.770 how they're thinking. 248 00:17:20.770 --> 00:17:24.240 So a student is admitted to a program as program offers 249 00:17:24.240 --> 00:17:26.080 particular courses. 250 00:17:26.080 --> 00:17:29.450 A program also includes milestones. 251 00:17:29.450 --> 00:17:34.450 And so this is the format we felt captured as simply as 252 00:17:35.590 --> 00:17:36.850 possible. 253 00:17:36.850 --> 00:17:40.590 The types of relationships that go on within a a PhD 254 00:17:40.590 --> 00:17:45.590 program, okay, so we can take this, 255 00:17:47.250 --> 00:17:52.250 this schema and we can use the syntax here that's called a 256 00:17:53.120 --> 00:17:58.120 cipher query to match or find all the paths that start from 257 00:17:59.560 --> 00:18:04.480 student and go one to four relationships away where they 258 00:18:04.480 --> 00:18:06.320 don't repeat. 259 00:18:06.320 --> 00:18:11.320 And where advised by doesn't relate back to a faculty member 260 00:18:14.020 --> 00:18:19.020 that yields this graph of 261 00:18:19.530 --> 00:18:24.530 9,620 paths, which is about 540 paths per student. 262 00:18:28.340 --> 00:18:32.160 We take those paths, this is sort of what a path looks like. 263 00:18:32.160 --> 00:18:35.570 You'll notice the all caps would be the relationships 264 00:18:35.570 --> 00:18:38.370 between different nodes. 265 00:18:39.810 --> 00:18:44.390 We take this text and then we turn it into natural language. 266 00:18:44.390 --> 00:18:46.223 We've got a JavaScript 267 00:18:47.590 --> 00:18:52.330 little module that that converts it into natural language 268 00:18:52.330 --> 00:18:57.330 because what we want to do is we want to pass this to a 269 00:18:59.760 --> 00:19:02.250 large language model to get embeddings. 270 00:19:02.250 --> 00:19:04.120 So this is what it turns into. 271 00:19:04.120 --> 00:19:08.010 So we've got the path that came from the graph database, 272 00:19:08.010 --> 00:19:11.280 we turn into natural language and then we turn that into 273 00:19:11.280 --> 00:19:12.470 embeddings. 274 00:19:12.470 --> 00:19:17.470 A lot of embeddings 1,536 per chunk of text 275 00:19:19.340 --> 00:19:20.173 per path. 276 00:19:22.750 --> 00:19:25.100 Okay, so what do we do with those embeddings? 277 00:19:25.100 --> 00:19:27.220 How do we store them, et cetera. 278 00:19:27.220 --> 00:19:30.890 And what are they? So you remember the large language model, 279 00:19:30.890 --> 00:19:35.120 we can input text and we can output the representation of 280 00:19:35.120 --> 00:19:37.610 that text as an embedding, 281 00:19:37.610 --> 00:19:40.730 as a numeric representation as shown before. 282 00:19:40.730 --> 00:19:45.730 So it must be a multiple of eight. You could have a, 283 00:19:46.980 --> 00:19:50.900 you know, a vector embedding of, you know, 16 numbers, 284 00:19:50.900 --> 00:19:53.520 but it's not gonna be very good. 285 00:19:53.520 --> 00:19:56.120 1,536 is a lot. 286 00:19:57.290 --> 00:19:58.070 However, 287 00:19:58.070 --> 00:20:02.920 that's what OpenAI has set their embeddings model at. 288 00:20:04.020 --> 00:20:09.020 Okay? So the idea is, is that you take each of your paths, 289 00:20:09.380 --> 00:20:10.660 each, you know, 290 00:20:10.660 --> 00:20:14.640 path through your content and turn it into an embedding. 291 00:20:14.640 --> 00:20:18.310 So you might have one that ends on the dissertation process, 292 00:20:18.310 --> 00:20:20.970 another on a comps exam process, 293 00:20:20.970 --> 00:20:25.360 and a third on a course 294 00:20:27.390 --> 00:20:30.340 and the details about the course. 295 00:20:30.340 --> 00:20:34.910 So then when you get a question, when a query comes in, 296 00:20:34.910 --> 00:20:39.910 you can turn that query into an embedding itself and then 297 00:20:42.370 --> 00:20:46.370 you can run something called cosigned similarity 298 00:20:47.580 --> 00:20:49.780 to figure out what's close to it. 299 00:20:49.780 --> 00:20:53.680 So here's an example where the query's kitten, 300 00:20:53.680 --> 00:20:58.500 and it's going into this vector space and the closest 301 00:20:58.500 --> 00:21:00.940 vectors, the close vectors are about cats, 302 00:21:00.940 --> 00:21:03.400 which you would expect. 303 00:21:03.400 --> 00:21:06.100 Now ours is a lot more complex, right? 304 00:21:06.100 --> 00:21:09.960 We've got coursework in here. So the word course coursework, 305 00:21:09.960 --> 00:21:12.980 you know, but they're saying after coursework. 306 00:21:12.980 --> 00:21:14.740 So there's a lot going on here. 307 00:21:14.740 --> 00:21:17.240 We're beyond cats and bananas, 308 00:21:20.110 --> 00:21:25.110 but we run the cosigned similarity and we get back values 309 00:21:25.230 --> 00:21:26.970 and then whatever those values are, 310 00:21:26.970 --> 00:21:30.930 we rank them against each other and then we look at, 311 00:21:30.930 --> 00:21:31.290 you know, 312 00:21:31.290 --> 00:21:36.290 which one was the best match and how high of a match was it? 313 00:21:36.850 --> 00:21:39.450 Because maybe it's the best match, but you know, 314 00:21:39.450 --> 00:21:43.170 it's still low, something has to be the best match. 315 00:21:43.170 --> 00:21:43.930 So if it's low, 316 00:21:43.930 --> 00:21:47.950 then maybe we don't want to use it as we go forward. 317 00:21:47.950 --> 00:21:49.130 Okay? 318 00:21:49.130 --> 00:21:53.450 So if we've got those, all those vectors, 319 00:21:53.450 --> 00:21:57.440 where do we store them previously we stored them in a 320 00:21:57.440 --> 00:21:58.700 JavaScript array, 321 00:21:58.700 --> 00:22:03.360 but eventually that's not gonna work because it's gonna bog 322 00:22:03.360 --> 00:22:05.850 the browser down or it's gonna choke off your server. 323 00:22:05.850 --> 00:22:10.640 So using a vector database is a, 324 00:22:14.320 --> 00:22:17.240 is a much faster, much more efficient way to go. 325 00:22:17.240 --> 00:22:20.580 So if we take those vectors we had before, 326 00:22:20.580 --> 00:22:25.580 once they're generated and we add them to a vector database, 327 00:22:26.830 --> 00:22:30.510 the one we're using is called pine cone, 328 00:22:30.510 --> 00:22:34.550 we can add metadata including the natural text that the 329 00:22:34.550 --> 00:22:38.080 vectors represent. So these are the, 330 00:22:38.080 --> 00:22:40.700 this is where the vectors go. And then, 331 00:22:40.700 --> 00:22:45.700 so this would be one vector with natural text 332 00:22:47.370 --> 00:22:50.060 associated with it, and then another and another. 333 00:22:50.060 --> 00:22:51.370 So 9,000, 334 00:22:51.370 --> 00:22:56.100 I don't know what we had 45 or whatever would all be 335 00:22:56.100 --> 00:23:01.100 populated into this vector space and get ingested in. 336 00:23:03.210 --> 00:23:07.470 Okay, how do vectors, oops, sorry. 337 00:23:07.470 --> 00:23:12.130 How do vector databases connect with neurology? 338 00:23:13.110 --> 00:23:14.780 So maybe again, 339 00:23:14.780 --> 00:23:18.220 is useful to align them and think about how they are 340 00:23:18.220 --> 00:23:19.280 aligned. 341 00:23:19.280 --> 00:23:23.230 One of the ways is high dimensional representations. 342 00:23:23.230 --> 00:23:27.300 So just like in the brain, 343 00:23:27.300 --> 00:23:31.780 there's complex neural activity patterns in this vector 344 00:23:31.780 --> 00:23:35.420 space. I mean, think about 1,536 dimensions. 345 00:23:35.420 --> 00:23:38.900 We can think about 2D and 3D pretty easily, 346 00:23:38.900 --> 00:23:42.740 but even thinking about 12 D is, you know, 347 00:23:43.610 --> 00:23:46.380 more than we can do without a lot of coffee. 348 00:23:46.380 --> 00:23:51.320 And so 1536 is fairly, fairly complex. 349 00:23:53.590 --> 00:23:54.080 Also, 350 00:23:54.080 --> 00:23:57.120 we've got distributed representations happening in both. 351 00:23:57.120 --> 00:23:59.320 A single neuron can play multiple roles, 352 00:23:59.320 --> 00:24:04.320 just like a single dimension within a vector can contribute 353 00:24:05.730 --> 00:24:08.397 to that vector in multiple ways. 354 00:24:13.010 --> 00:24:13.380 Additionally, 355 00:24:13.380 --> 00:24:18.380 there's both use aspects of similarity based retrieval. 356 00:24:18.920 --> 00:24:20.810 Just like, you know, someone, you know, 357 00:24:20.810 --> 00:24:24.130 you light a match to start the grill and that smell of smoke 358 00:24:24.130 --> 00:24:28.730 might trigger a memory from being at the lake the summer 359 00:24:28.730 --> 00:24:29.563 before. 360 00:24:30.700 --> 00:24:35.530 Similar with that cat and kitten idea or you know, 361 00:24:35.530 --> 00:24:38.770 this cosigned similarity matching. 362 00:24:38.770 --> 00:24:42.603 What's closer together is what gets retrieved. 363 00:24:43.730 --> 00:24:47.820 Okay, again, it's not a perfect connection. 364 00:24:47.820 --> 00:24:50.140 There's more complexity in the brain obviously, 365 00:24:50.140 --> 00:24:53.660 and different data types with both as well as different 366 00:24:53.660 --> 00:24:54.577 mechanisms. 367 00:24:55.600 --> 00:24:58.870 Okay, design. 368 00:25:00.780 --> 00:25:05.780 So in terms of data, the data prepping workflow, 369 00:25:05.870 --> 00:25:10.810 how do we get everything ready? Like before users show up, 370 00:25:10.810 --> 00:25:15.810 we start with a bunch of spreadsheets about the program. 371 00:25:16.880 --> 00:25:20.080 Those get converted into CSV files, 372 00:25:20.080 --> 00:25:24.400 which then get ingested into the graphing database, 373 00:25:24.400 --> 00:25:29.400 which then generates all the paths that exist within it that 374 00:25:30.110 --> 00:25:31.350 are relevant. 375 00:25:31.350 --> 00:25:33.210 And then those paths, 376 00:25:33.210 --> 00:25:37.010 each one is turned into natural text and sent to become an 377 00:25:37.010 --> 00:25:37.710 embedding. 378 00:25:37.710 --> 00:25:42.710 And then those embeddings are stored in the vector database. 379 00:25:44.700 --> 00:25:49.120 And then on the, so the advising bot workflow, 380 00:25:50.110 --> 00:25:55.110 once someone logs in, they ask, they, you know, 381 00:25:55.880 --> 00:25:58.000 ask a question, they make a statement, 382 00:25:58.000 --> 00:25:58.760 whatever they're gonna do. 383 00:25:58.760 --> 00:26:03.010 And then that query is converted into an embedding. 384 00:26:04.230 --> 00:26:09.230 And that embedding is, you know, 385 00:26:09.700 --> 00:26:14.700 sent to the vector database and compared a score is brought 386 00:26:15.030 --> 00:26:18.630 back as well as the natural text associated with that 387 00:26:18.630 --> 00:26:19.690 vector. 388 00:26:19.690 --> 00:26:23.100 And if it's above a threshold score, there's no magic score. 389 00:26:23.100 --> 00:26:25.150 I put 0.7 here, 390 00:26:25.150 --> 00:26:30.150 we're still working on what our threshold is going to be, 391 00:26:30.550 --> 00:26:32.620 but if it's lower than your threshold, 392 00:26:32.620 --> 00:26:37.620 then our approach is to send the query for open-ended chat 393 00:26:38.270 --> 00:26:39.770 response. 394 00:26:39.770 --> 00:26:42.320 But if it's above the the threshold, 395 00:26:42.320 --> 00:26:46.640 then we are sending both the query and the tout matching 396 00:26:46.640 --> 00:26:51.640 content to open AI to generate a response based on the 397 00:26:52.440 --> 00:26:56.440 content and then the student receives the reply. 398 00:27:00.380 --> 00:27:01.830 Okay, Hector, go ahead, 399 00:27:01.830 --> 00:27:05.497 just tell me when to move the slide forward. 400 00:27:12.600 --> 00:27:14.010 You'll have to unmute. 401 00:27:14.010 --> 00:27:17.340 Hector. Oh yes, now I'm ready. 402 00:27:17.340 --> 00:27:18.173 Go for it. 403 00:27:20.730 --> 00:27:22.830 Okay, we're work, I'm working right now. 404 00:27:22.830 --> 00:27:26.060 We are working in the desktop design, 405 00:27:27.340 --> 00:27:32.340 basically studying, can we move to the 68? 406 00:27:34.940 --> 00:27:39.940 Why do we call it oria? Is this I thing as mispronunciation? 407 00:27:40.780 --> 00:27:44.540 The first idea was to call it jic, 408 00:27:44.540 --> 00:27:49.540 like the hamlet skull that represents the life afterlife, 409 00:27:50.880 --> 00:27:55.880 but I usually say Jor jy. So people was trying to say like, 410 00:27:57.140 --> 00:28:00.580 was like the dog, but actually was jurich not jerky. 411 00:28:00.580 --> 00:28:03.870 So then I decided to set ratio was better, 412 00:28:03.870 --> 00:28:08.220 so we decided to call it ratio. And it's not just a chatbot, 413 00:28:08.220 --> 00:28:09.460 that's why we. 414 00:28:09.460 --> 00:28:11.580 Sorry, Ector, sorry for interrupting. 415 00:28:11.580 --> 00:28:13.260 It's not Jurich, it's Jok. 416 00:28:13.260 --> 00:28:14.020 Sorry. 417 00:28:14.020 --> 00:28:15.150 Jaric. 418 00:28:15.150 --> 00:28:16.180 No, it's not Jurich. 419 00:28:16.180 --> 00:28:18.990 The correct is Yorick. He's Danish. 420 00:28:18.990 --> 00:28:20.220 Oh, ok. 421 00:28:20.220 --> 00:28:22.120 We're talking about Hamlet, right? 422 00:28:22.120 --> 00:28:22.820 Yes, yes. 423 00:28:22.820 --> 00:28:25.020 Yeah, there's no jurich there, believe me, 424 00:28:25.020 --> 00:28:29.100 there is Yorick in English. They twisted it. 425 00:28:29.100 --> 00:28:29.820 Okay. 426 00:28:29.820 --> 00:28:31.950 Yes. Oh, thank you. 427 00:28:31.950 --> 00:28:35.367 So because of that I decided to for Oria. 428 00:28:37.190 --> 00:28:41.490 So, and then because we have, 429 00:28:41.490 --> 00:28:46.490 we were discussing if a ratio was going to be embedded into 430 00:28:46.850 --> 00:28:51.850 this whole website or we would like to have just individual 431 00:28:52.080 --> 00:28:53.860 a ratio website. 432 00:28:53.860 --> 00:28:57.970 So that's why we have can you go to the 68 1? 433 00:28:59.780 --> 00:29:02.060 So we, we I, 434 00:29:02.060 --> 00:29:04.470 we were discussing about the interaction design, 435 00:29:04.470 --> 00:29:06.940 how we would like the, 436 00:29:06.940 --> 00:29:11.450 or be interacting with the graduate students, 437 00:29:12.290 --> 00:29:16.120 what the product strategy could be, what is the content, 438 00:29:16.120 --> 00:29:21.120 what the interaction design and also the I repeat 439 00:29:21.810 --> 00:29:24.730 interaction and the user research. What were the questions? 440 00:29:24.730 --> 00:29:28.290 So we're asking some of the students around the department 441 00:29:28.290 --> 00:29:32.050 what were the most common questions we would like to ask the 442 00:29:32.050 --> 00:29:36.290 advisors. And the questions were different according to the, 443 00:29:36.290 --> 00:29:39.770 the years we were studying the program. 444 00:29:40.660 --> 00:29:45.440 So on that we have this design. The first one was, 445 00:29:46.540 --> 00:29:51.160 can we go to the 69? So it was that just the basic design, 446 00:29:51.160 --> 00:29:53.160 just a small one. 447 00:29:53.160 --> 00:29:54.840 We want just the students to come here. 448 00:29:54.840 --> 00:29:57.640 We have some prompts so the students can choose what they 449 00:29:57.640 --> 00:30:02.640 want according to the level of the program they are studying 450 00:30:02.970 --> 00:30:04.860 if they are PhD or masters. 451 00:30:04.860 --> 00:30:07.800 But then we decided that we don't need that because we have 452 00:30:07.800 --> 00:30:10.240 the data and we know what the students are and the program 453 00:30:10.240 --> 00:30:13.400 they are right now. So we, 454 00:30:13.400 --> 00:30:16.280 we are going to include o other questions so the students 455 00:30:16.280 --> 00:30:19.470 can actually choose from those questions. So we, 456 00:30:19.470 --> 00:30:23.120 I wanted the oria to have like a more human 457 00:30:26.530 --> 00:30:27.750 figure. 458 00:30:27.750 --> 00:30:31.340 So that's why you can see that figure person there, 459 00:30:31.340 --> 00:30:35.780 like Oria is like a more graduate student look like. 460 00:30:36.780 --> 00:30:41.580 And then also that's gonna be also the answer that the oria 461 00:30:41.580 --> 00:30:43.530 is gonna give more friendly, 462 00:30:43.530 --> 00:30:47.900 it's gonna be like kind of another path for the grad 463 00:30:47.900 --> 00:30:51.670 students. And if when they oria doesn't know the answer, 464 00:30:51.670 --> 00:30:56.580 he will actually guide the grad students to contact their 465 00:30:56.580 --> 00:31:00.540 own advisor. So they, 466 00:31:00.540 --> 00:31:03.800 this is a Jesus actually how the, 467 00:31:03.800 --> 00:31:08.580 the colors and the design will be in the actual website and 468 00:31:08.580 --> 00:31:10.460 then the 70 will show, 469 00:31:12.830 --> 00:31:16.490 the final decision was to have just one website for the 470 00:31:16.490 --> 00:31:18.190 chatbot for Oria. 471 00:31:18.190 --> 00:31:21.810 And they people can come here and they will can then also 472 00:31:21.810 --> 00:31:26.650 find some milestone information or some information about 473 00:31:26.650 --> 00:31:31.650 conferences or research styles and information about any 474 00:31:32.570 --> 00:31:34.950 other information important for grad students. 475 00:31:34.950 --> 00:31:39.867 And also they can interact with oration. So that's how the, 476 00:31:41.230 --> 00:31:46.147 the design was was taught and we are still working on that. 477 00:31:47.070 --> 00:31:51.080 Great. Do you wanna mention about Horatio's T-shirt? 478 00:31:51.080 --> 00:31:56.080 Oh yeah. Hario T-shirt is actually hello in Arabic. 479 00:31:58.090 --> 00:32:03.090 But actually if you go and see closer it looks like two eyes 480 00:32:03.610 --> 00:32:06.990 and I just like a smile. It looks like it's a smiling, 481 00:32:06.990 --> 00:32:10.200 but actually it's an Arabic board that means hello. 482 00:32:10.200 --> 00:32:15.200 So it's like it's actually greeting the students in Arabic. 483 00:32:15.950 --> 00:32:17.990 Great, thanks Hector. 484 00:32:17.990 --> 00:32:18.823 Thank you. 485 00:32:20.330 --> 00:32:25.270 All right. So on slide 71, 486 00:32:25.270 --> 00:32:28.800 we've got some of the, 487 00:32:28.800 --> 00:32:32.910 so these would be the iterative demos and I didn't put bot 488 00:32:32.910 --> 00:32:35.310 zero or bot zero one. 489 00:32:35.310 --> 00:32:39.820 I skipped a few because there wasn't a ton that changed or 490 00:32:39.820 --> 00:32:42.820 you know, it started out real rough. 491 00:32:43.700 --> 00:32:48.390 But as they go up, the first one is just G P T three, 492 00:32:48.390 --> 00:32:53.390 a simple response bot three includes a prompt to angle it 493 00:32:53.800 --> 00:32:58.800 toward advising and then six includes a summarizer and then 494 00:32:59.430 --> 00:33:02.920 part of the ui ux design that Hector led. 495 00:33:03.760 --> 00:33:08.200 It was integrated in the current version of Horacio. 496 00:33:10.500 --> 00:33:15.500 And then to show what it looks like with prompted with 497 00:33:15.960 --> 00:33:17.530 embeddings, 498 00:33:17.530 --> 00:33:22.440 we have a prototype called Alex that's for a particular 499 00:33:22.440 --> 00:33:23.810 course. 500 00:33:23.810 --> 00:33:28.350 And so it will answer questions about that course with 501 00:33:28.350 --> 00:33:31.390 course content. And usually if it doesn't know, 502 00:33:31.390 --> 00:33:34.280 it'll just say, sorry, I don't have that information. 503 00:33:34.280 --> 00:33:39.280 Whereas earlier versions would just make anything up what 504 00:33:39.910 --> 00:33:44.910 open AI calls hallucinating, sort of like, 505 00:33:46.610 --> 00:33:47.443 you know, 506 00:33:48.980 --> 00:33:53.080 you probably know someone who would rather say something and 507 00:33:53.080 --> 00:33:57.200 look smart than say they don't know that's what the chatbot 508 00:33:57.200 --> 00:33:58.440 used to do. 509 00:33:58.440 --> 00:34:01.680 3.5 is a little better in that regard, 510 00:34:01.680 --> 00:34:06.240 especially if you send it, if you add embeddings to it. 511 00:34:06.240 --> 00:34:10.230 And then what we're working toward now is G P T four 512 00:34:10.230 --> 00:34:14.320 prompted with embeddings for orio 513 00:34:20.010 --> 00:34:21.010 future work, 514 00:34:23.230 --> 00:34:28.230 our interest is in sort of leveraging some of the, 515 00:34:28.250 --> 00:34:33.250 some of the connections between the brain and the way data 516 00:34:33.370 --> 00:34:36.580 is stored, especially in graphing databases. 517 00:34:36.580 --> 00:34:38.970 So users, 518 00:34:38.970 --> 00:34:43.050 students in this case can have interactions with the bot and 519 00:34:43.050 --> 00:34:46.880 then we send the bot back through and say, what, 520 00:34:46.880 --> 00:34:51.880 what new connections can you add to the graphing data or the 521 00:34:53.210 --> 00:34:57.670 graph database based on the conversation you had. 522 00:34:57.670 --> 00:35:00.490 And then two in the morning, 523 00:35:00.490 --> 00:35:05.490 four in the morning at some point run those in order for it 524 00:35:05.770 --> 00:35:09.290 to go maybe from short term to long term memory. 525 00:35:09.290 --> 00:35:11.790 So it gets added, paths get generated, 526 00:35:11.790 --> 00:35:14.170 new embeddings for those paths get made. 527 00:35:14.170 --> 00:35:17.170 And so those could be, you know, 528 00:35:17.170 --> 00:35:21.890 acted on or brought back as part of the content so it could 529 00:35:21.890 --> 00:35:22.590 remember things. 530 00:35:22.590 --> 00:35:27.170 And then farther out beyond this project would be learning 531 00:35:27.170 --> 00:35:31.410 based graphs and vector name spaces that that might go into, 532 00:35:31.410 --> 00:35:31.650 you know, 533 00:35:31.650 --> 00:35:34.080 how they're thinking about different topics within, 534 00:35:34.080 --> 00:35:37.760 within academia, within getting their PhD, 535 00:35:37.760 --> 00:35:41.593 within their dissertation research, et cetera. 536 00:35:44.260 --> 00:35:46.510 We'd like to thank the U N L Center for Science, 537 00:35:46.510 --> 00:35:50.510 math and Computer Science ed for the support they've given 538 00:35:50.510 --> 00:35:51.280 us, 539 00:35:51.280 --> 00:35:56.280 as well as open AI for early G P T and plugin development 540 00:35:56.390 --> 00:36:00.390 access and NEO four J for letting us into their academic 541 00:36:00.390 --> 00:36:02.807 research and startup program. 542 00:36:03.820 --> 00:36:08.470 Some references and it's time for the q and a. 543 00:36:08.470 --> 00:36:12.070 We can hang out on the iterative demos if you've got 544 00:36:12.070 --> 00:36:14.510 questions or I'll, better yet, 545 00:36:14.510 --> 00:36:19.060 I'll just stop talking and if there are questions we can 546 00:36:19.060 --> 00:36:20.310 talk about 'em. 547 00:36:23.640 --> 00:36:28.240 Thank you very much. I really enjoyed that presentation. 548 00:36:28.240 --> 00:36:31.470 Does anyone have any questions? Anyone from the audience? 549 00:36:31.470 --> 00:36:32.400 I figured I'd ask, 550 00:36:32.400 --> 00:36:36.567 let other people ask first before I bring up mine. 551 00:36:37.960 --> 00:36:41.377 Yes, I, I was very interesting Bruce bra, 552 00:36:42.930 --> 00:36:46.500 a lot of the data that we have about things like courses and 553 00:36:46.500 --> 00:36:50.340 so forth is fairly structured and can fit in like an 554 00:36:50.340 --> 00:36:52.690 ontology base. 555 00:36:52.690 --> 00:36:57.140 I, is there a way to kind of do a hybrid of the tech and the 556 00:36:57.140 --> 00:36:59.460 ontologies knowing that they, 557 00:36:59.460 --> 00:37:03.960 they fit in graph databases pretty well? Sometimes. 558 00:37:03.960 --> 00:37:08.960 Yes, there are ways to automate the ingestion into a graph 559 00:37:10.500 --> 00:37:15.030 database and it seems like, you know, 560 00:37:15.030 --> 00:37:19.900 if plugins sort of becomes a thing that's open to a wider 561 00:37:19.900 --> 00:37:20.700 audience and you know, 562 00:37:20.700 --> 00:37:24.620 there are understandable risks and reasons why that that 563 00:37:24.620 --> 00:37:28.537 might, there might be a need for pause on that. 564 00:37:29.600 --> 00:37:34.220 But, but yes, we can, you could just take the, 565 00:37:34.220 --> 00:37:39.220 the info you had the whole, the whole reason of, you know, 566 00:37:39.300 --> 00:37:44.300 turning things into an embedding is because that the plug-in 567 00:37:44.940 --> 00:37:49.700 option is not out, is not an option just out of the box. 568 00:37:50.580 --> 00:37:53.260 They're still figuring out who should be able to do it and 569 00:37:53.260 --> 00:37:55.460 they're vetting which, 570 00:37:55.460 --> 00:37:59.620 which plug-ins to release to the public in the early plug-in 571 00:37:59.620 --> 00:38:00.640 stage. 572 00:38:00.640 --> 00:38:05.640 You can have other plug-in developers try out your plugin, 573 00:38:05.720 --> 00:38:07.660 but, but you know, you can't, 574 00:38:07.660 --> 00:38:09.650 it can't be released to the public. 575 00:38:09.650 --> 00:38:11.510 And so there's a, 576 00:38:12.380 --> 00:38:17.350 I can see a way where you might not even need to go the 577 00:38:17.350 --> 00:38:19.620 route we went a couple months from now, 578 00:38:19.620 --> 00:38:22.470 a couple years from now, a couple of weeks from now. 579 00:38:22.470 --> 00:38:24.050 But for now, 580 00:38:24.050 --> 00:38:28.350 the way to do it seems to be with the graphing database and 581 00:38:28.350 --> 00:38:32.680 then the pathways through it, turn those into embeddings. 582 00:38:32.680 --> 00:38:36.960 Someone laughed also follow up on that if it was meant for 583 00:38:36.960 --> 00:38:38.600 this, for this dog. 584 00:38:51.750 --> 00:38:54.030 Actually I was, I just wanted to not, 585 00:38:54.030 --> 00:38:56.370 not anything to do with any laughter, 586 00:38:56.370 --> 00:39:00.500 but the talking about the embeddings, 587 00:39:00.500 --> 00:39:05.500 you could arguably exercise more control over the responses 588 00:39:06.640 --> 00:39:08.880 that the agent gives by, through, 589 00:39:08.880 --> 00:39:12.490 through the embeddings instead of leaving it wide open. 590 00:39:12.490 --> 00:39:14.400 You know, that big part of what, 591 00:39:14.400 --> 00:39:19.170 what's been getting publicity is prompt engineering to try 592 00:39:19.170 --> 00:39:24.050 to get chat g p t to say very inappropriate things in a 593 00:39:24.050 --> 00:39:27.170 variety of contexts. And, and you know, 594 00:39:27.170 --> 00:39:29.610 this is the kind of thing that that should be done because, 595 00:39:29.610 --> 00:39:32.010 because they need to enhance the filters and so on. 596 00:39:32.010 --> 00:39:34.130 But we've been, 597 00:39:34.130 --> 00:39:36.810 we're looking at a couple of projects that are gonna try to 598 00:39:36.810 --> 00:39:41.810 use LLMs kind of cooperating with some kind of a graphical 599 00:39:42.460 --> 00:39:45.970 data representation, like maybe a base net or something. 600 00:39:45.970 --> 00:39:50.970 And my my hope is that if we're focused on like if, if the, 601 00:39:52.790 --> 00:39:57.790 if the responses are kinda rooted somehow in the database, 602 00:39:58.300 --> 00:39:59.660 the knowledge base that we're using, 603 00:39:59.660 --> 00:40:01.250 kinda like what you're doing, 604 00:40:01.250 --> 00:40:04.010 I think that you can cross your fingers, you know, 605 00:40:04.010 --> 00:40:06.540 hope that it and expect that it would not, 606 00:40:06.540 --> 00:40:09.620 it'd be much less likely to be able to be tricked into just 607 00:40:09.620 --> 00:40:13.220 saying something that's completely awful without having to 608 00:40:13.220 --> 00:40:15.460 do additional filters. And by like, 609 00:40:15.460 --> 00:40:17.380 like when you're talking about using the similarity, 610 00:40:17.380 --> 00:40:22.380 if someone enters a prompt that is completely, you know, 611 00:40:22.880 --> 00:40:25.860 far and away from anything in your embedded space, 612 00:40:25.860 --> 00:40:29.870 then you know it's at least not appropriate for the context. 613 00:40:29.870 --> 00:40:32.100 It might not be malicious but it might not just, 614 00:40:32.100 --> 00:40:34.220 it might just not be appropriate for this thing to answer. 615 00:40:34.220 --> 00:40:36.340 And so you can just say, sorry, I don't know. 616 00:40:36.340 --> 00:40:38.220 And maybe forward it to a human or something. 617 00:40:38.220 --> 00:40:40.700 So I think that I, I understand you know, 618 00:40:40.700 --> 00:40:42.610 what you're talking about and I think there's, 619 00:40:42.610 --> 00:40:45.460 I would be interested to see how well in a controlled 620 00:40:45.460 --> 00:40:45.900 experiment, 621 00:40:45.900 --> 00:40:50.900 how well chat G P t especially version four would fare just 622 00:40:51.020 --> 00:40:53.850 on its own for advising. But I'm, 623 00:40:55.030 --> 00:40:58.560 I I I think that there might be actual value in using your 624 00:40:58.560 --> 00:41:00.800 knowledge base cause then you can adjust the knowledge base 625 00:41:00.800 --> 00:41:05.800 as you want, expand it or, or throw things out as necessary. 626 00:41:05.970 --> 00:41:06.650 For sure. 627 00:41:06.650 --> 00:41:10.990 And there's a system message that that's baked into our 628 00:41:10.990 --> 00:41:13.170 prompt, even if it's open-ended, 629 00:41:13.170 --> 00:41:16.250 the system message says you're a PhD advisor, 630 00:41:16.250 --> 00:41:21.250 you are giving feedback to a PhD student and stay in that 631 00:41:21.630 --> 00:41:23.713 domain. They also have a, 632 00:41:26.750 --> 00:41:28.600 they have a way where you can send their, 633 00:41:28.600 --> 00:41:31.860 the prompt that you receive to make sure that it's, 634 00:41:31.860 --> 00:41:34.400 you know, that it's gonna be ok. 635 00:41:34.400 --> 00:41:35.640 There was, 636 00:41:35.640 --> 00:41:40.640 Lex Freeman had Sam Altman on his podcast a couple weeks ago 637 00:41:40.900 --> 00:41:45.900 and this is the c e o of open ai and he was saying that, 638 00:41:46.180 --> 00:41:48.640 you know, when people are getting these, 639 00:41:48.640 --> 00:41:51.870 these responses that are objectionable, 640 00:41:51.870 --> 00:41:56.870 it's usually around the 5000th response to the same question 641 00:41:56.950 --> 00:42:00.720 that something odd comes up in the 10000th response to the 642 00:42:00.720 --> 00:42:03.440 same question where, you know, something, you know, 643 00:42:03.440 --> 00:42:07.100 that that goes viral happens. 644 00:42:07.100 --> 00:42:09.200 And so I think with R L H F, 645 00:42:09.200 --> 00:42:13.200 I think that's gonna go down and I agree with you that I 646 00:42:13.200 --> 00:42:16.000 think there are lots of levels of things that, 647 00:42:16.000 --> 00:42:19.880 that you can do. So i, I take responsibility for, 648 00:42:19.880 --> 00:42:23.200 for writing open-ended instead of, you know, 649 00:42:23.200 --> 00:42:25.120 still saying it's still within the domain, 650 00:42:25.120 --> 00:42:29.500 it's still within what would be, you know, appropriate, 651 00:42:29.500 --> 00:42:30.710 but it's not, 652 00:42:30.710 --> 00:42:35.710 it's not being sort of constrained to text because 653 00:42:37.680 --> 00:42:41.600 when we send it the text we say base your response on the 654 00:42:41.600 --> 00:42:46.600 following text and then we send it that the, you know, 655 00:42:46.680 --> 00:42:51.620 the embedding equivalent natural text that it matched with. 656 00:42:51.620 --> 00:42:55.320 And so we're just removing that part. 657 00:42:57.120 --> 00:42:57.953 Okay. 658 00:42:58.900 --> 00:43:00.440 And, and ju just, 659 00:43:00.440 --> 00:43:02.640 I don't wanna take away time for my other questions, 660 00:43:02.640 --> 00:43:05.170 but just very briefly, 661 00:43:05.170 --> 00:43:09.280 do you see any major issues that would come up if say the 662 00:43:09.280 --> 00:43:12.680 underlying graphical knowledge base was probabilistic like 663 00:43:12.680 --> 00:43:16.950 aine network as opposed to the strict graph representation, 664 00:43:16.950 --> 00:43:21.950 like the ontological kind of view that you are using? 665 00:43:22.030 --> 00:43:27.030 I I'm not sure I know enough about the difference between 666 00:43:28.040 --> 00:43:31.450 them to, to say you might have a better handle on that. 667 00:43:31.450 --> 00:43:34.800 We just, you know, we just have, 668 00:43:34.800 --> 00:43:39.040 and we haven't done a lot with strengths yet or how to set 669 00:43:39.040 --> 00:43:39.720 those and, 670 00:43:39.720 --> 00:43:44.720 and how to have those influence influence things in ours. 671 00:43:47.370 --> 00:43:50.620 Sounds like you're farther along in that thinking than, 672 00:43:50.620 --> 00:43:52.203 than we are so far. 673 00:43:53.310 --> 00:43:55.790 Not really, which is kinda why I asked the question. 674 00:43:55.790 --> 00:43:59.430 But I mean we were, we were actually trying to, 675 00:43:59.430 --> 00:44:01.710 we're still in the early stages and we were trying to think 676 00:44:01.710 --> 00:44:05.510 of ways to combine some graphical representation of a 677 00:44:05.510 --> 00:44:08.110 knowledge base in this case probabilistic just cause of the 678 00:44:08.110 --> 00:44:10.610 healthcare application angle. 679 00:44:10.610 --> 00:44:14.030 And one thing that we were actually kicking around was just 680 00:44:14.030 --> 00:44:18.950 feeding the graphical model in as part of the input in a 681 00:44:18.950 --> 00:44:23.950 multimodal model and, and asking it, okay, you know, are, 682 00:44:24.970 --> 00:44:26.230 you know, asking the model, 683 00:44:26.230 --> 00:44:27.950 can you answer questions about this? 684 00:44:27.950 --> 00:44:30.150 And then seeing where that goes. But I actually, 685 00:44:30.150 --> 00:44:33.120 I like your path based approach better. 686 00:44:33.120 --> 00:44:34.760 It seems a little bit more like, 687 00:44:34.760 --> 00:44:38.150 like you can control what the result's gonna be. 688 00:44:38.150 --> 00:44:42.330 Yeah. What we found at the first time we did this, we had, 689 00:44:42.330 --> 00:44:44.960 we sort of went node heavy, or sorry, 690 00:44:44.960 --> 00:44:48.470 we went node sparse and we had a lot of properties 691 00:44:48.470 --> 00:44:53.470 associated with each node and we jammed a bunch of, we, we, 692 00:44:53.530 --> 00:44:58.530 we had a lot of of information in the nodes and then we 693 00:44:59.320 --> 00:45:01.040 realized that that wasn't the way to do it. 694 00:45:01.040 --> 00:45:05.690 What we wanted was sort of sparse sparsely pop, 695 00:45:05.690 --> 00:45:09.800 we wanted sparsely parameter nodes that, 696 00:45:09.800 --> 00:45:12.160 that had specific information than just, 697 00:45:12.160 --> 00:45:16.280 we just had more nodes and more paths and that ended up 698 00:45:16.280 --> 00:45:18.120 being better because of the, you know, the, 699 00:45:18.120 --> 00:45:23.120 the maximum the way you max out on tokens going in and you 700 00:45:23.360 --> 00:45:26.760 know, if a student is asking about, you know, the, 701 00:45:26.760 --> 00:45:30.800 what the course attendance policy is to reference the Alex 702 00:45:30.800 --> 00:45:31.633 bot, 703 00:45:32.470 --> 00:45:35.760 they don't need the description of the course to come along 704 00:45:35.760 --> 00:45:40.760 with it. And so we've sort of, you know, even though our, 705 00:45:41.020 --> 00:45:46.020 our schema seem is seems pretty modest in terms of, 706 00:45:48.450 --> 00:45:52.390 you know, how many different types of nodes we have, 707 00:45:52.390 --> 00:45:57.390 we try to have not as like less information associated with 708 00:45:58.350 --> 00:46:00.440 each course so that, 709 00:46:00.440 --> 00:46:02.510 so that we we're running more efficiently. 710 00:46:02.510 --> 00:46:06.470 So that path that comes back is a little more efficient. 711 00:46:06.470 --> 00:46:09.350 And I think part of what we're gonna figure out or what 712 00:46:09.350 --> 00:46:11.390 we're we're gonna find is that the, 713 00:46:11.390 --> 00:46:15.350 like more relationships is better sort of like, you know, 714 00:46:15.350 --> 00:46:18.790 the brain if there are what thousand, 7,000, 715 00:46:18.790 --> 00:46:21.750 8,000 connections between, you know, 716 00:46:21.750 --> 00:46:23.833 one neuron is headed out. 717 00:46:24.890 --> 00:46:26.650 Not to say there's an average, but, 718 00:46:26.650 --> 00:46:29.230 but what we have now I think we're gonna find is we don't 719 00:46:29.230 --> 00:46:33.710 have enough relationships and figuring out how to sort of 720 00:46:33.710 --> 00:46:38.550 turn up the relationships might be a way to be able to bring 721 00:46:38.550 --> 00:46:43.080 back just the information that's related to, to the, 722 00:46:43.080 --> 00:46:44.640 the user query. 723 00:46:48.180 --> 00:46:50.680 All right. Well now I feel like I've monopolized your time. 724 00:46:50.680 --> 00:46:53.160 Is anyone else have other questions? 725 00:46:53.160 --> 00:46:56.243 Cause we still have about 10 minutes. 726 00:47:12.220 --> 00:47:14.137 I think is back to you. 727 00:47:15.450 --> 00:47:18.390 I guess so. Well the only other thing is just me, 728 00:47:18.390 --> 00:47:21.890 me hoping out loud that you limit this to, 729 00:47:22.980 --> 00:47:26.250 to with the academic advising and not the research advising. 730 00:47:26.250 --> 00:47:28.090 Cause there's some loads I'd like, 731 00:47:28.090 --> 00:47:30.880 there's some jobs I would like, like to have, 732 00:47:30.880 --> 00:47:32.000 have taken away from me, 733 00:47:32.000 --> 00:47:36.030 but not necessarily the research advising. So. 734 00:47:36.030 --> 00:47:37.000 Yeah. Have you, 735 00:47:37.000 --> 00:47:41.210 have you played around with G P T four in terms of, 736 00:47:41.210 --> 00:47:44.400 of self-research advising? Not to say that you need it, 737 00:47:44.400 --> 00:47:47.740 but I've played around with a little bit and it feels like, 738 00:47:47.740 --> 00:47:50.830 you know, someone to think with or a, a. 739 00:47:50.830 --> 00:47:51.120 A. 740 00:47:51.120 --> 00:47:52.787 Thing to think with. 741 00:47:53.850 --> 00:47:58.850 I, I, I, so, so one of my kids is very big on, on, 742 00:47:59.470 --> 00:48:04.300 on ll playing with LLMs and has been especially playing with 743 00:48:04.300 --> 00:48:07.300 G P T four and actually has generate, 744 00:48:07.300 --> 00:48:11.700 has made it generate study guides for exams coming up and 745 00:48:11.700 --> 00:48:13.800 now like, on a variety of topics. 746 00:48:13.800 --> 00:48:15.740 And so yeah. Yeah, I mean it's, 747 00:48:15.740 --> 00:48:18.100 and I haven't played with that aspect of it myself, 748 00:48:18.100 --> 00:48:19.860 but it does sound very useful and then, 749 00:48:19.860 --> 00:48:23.080 and a little bit terrifying I suppose. 750 00:48:23.080 --> 00:48:23.500 Yes. 751 00:48:23.500 --> 00:48:27.110 And I think that that terror comes up in lots of different, 752 00:48:27.110 --> 00:48:32.110 in lots of different areas from coding to like you said, 753 00:48:32.140 --> 00:48:33.040 like, whoa, 754 00:48:33.040 --> 00:48:35.700 I'm really good at this thing that's called research 755 00:48:35.700 --> 00:48:39.260 advising and what if this thing can do it for me. 756 00:48:39.260 --> 00:48:40.730 I'm just gonna pull up a, 757 00:48:40.730 --> 00:48:44.540 a different presentation that we did where we were talking 758 00:48:44.540 --> 00:48:49.540 with secondary education teachers and we tried to have 759 00:48:51.500 --> 00:48:54.500 him think about a violinist and a conductor and getting 760 00:48:54.500 --> 00:48:57.560 their students to think of themselves more as a conductor 761 00:48:57.560 --> 00:49:00.060 and maybe ourselves as more of a conductor. 762 00:49:00.060 --> 00:49:02.790 You still have to know a ton about music, you know, 763 00:49:02.790 --> 00:49:05.430 to conduct and to get it to work. 764 00:49:05.430 --> 00:49:08.950 So I think even the research advising part, you know, 765 00:49:08.950 --> 00:49:13.750 to have them have a way to sort of bounce ideas off of, 766 00:49:13.750 --> 00:49:17.580 of something when those bottlenecks come up, you know, 767 00:49:17.580 --> 00:49:22.580 when there's not enough, when there's not enough of, 768 00:49:24.690 --> 00:49:28.510 of an advisor to go around to interact with, 769 00:49:28.510 --> 00:49:33.280 with a student when they need it or when they want it could 770 00:49:33.280 --> 00:49:36.920 be useful. We're, we are not trying to replace advising. 771 00:49:36.920 --> 00:49:41.220 We're, we're trying to, you know, take some of the, 772 00:49:41.220 --> 00:49:46.220 the peripheral administrativia logistical pieces and get, 773 00:49:48.200 --> 00:49:50.450 make those more accessible. 774 00:49:52.570 --> 00:49:54.320 So looks like we have oh, sorry. 775 00:49:54.320 --> 00:49:56.440 Yeah, we, we have some people who are unmuting now. 776 00:49:56.440 --> 00:49:56.880 So yeah. 777 00:49:56.880 --> 00:49:59.463 This isle I am, I'm at U N M C. 778 00:50:00.790 --> 00:50:03.160 I have never come up with this idea before, 779 00:50:03.160 --> 00:50:07.440 but this conversation brought me to that if we're replacing 780 00:50:07.440 --> 00:50:08.080 the advisor, 781 00:50:08.080 --> 00:50:12.663 shouldn't we also replace the students with Chad G P T? 782 00:50:15.240 --> 00:50:16.490 How about that? 783 00:50:18.440 --> 00:50:20.790 Th this of course is meant as a joke, 784 00:50:20.790 --> 00:50:23.920 but there is some truth in any joke. 785 00:50:23.920 --> 00:50:26.920 What do you think about this Hector? 786 00:50:30.260 --> 00:50:31.843 The question again? 787 00:50:34.090 --> 00:50:37.410 We're about to replace the, the advisors, the, 788 00:50:37.410 --> 00:50:38.970 the academic advisors, 789 00:50:38.970 --> 00:50:42.130 the research advisors with chat G P T or even more 790 00:50:42.130 --> 00:50:46.050 intelligent systems now how about replacing also the 791 00:50:46.050 --> 00:50:49.760 students with systems that are chat G P T based? 792 00:50:49.760 --> 00:50:53.710 Yeah, actually we are not thinking to replace them, 793 00:50:53.710 --> 00:50:58.530 but we are actually help the students with advisors not 794 00:50:58.530 --> 00:50:59.830 replace them. 795 00:50:59.830 --> 00:51:01.150 And I don't, 796 00:51:01.150 --> 00:51:04.930 and also have an idea that we are not against artificial 797 00:51:04.930 --> 00:51:09.930 intelligence, but we can improve our humanity. 798 00:51:10.080 --> 00:51:14.413 We are humans and we can improve their tenderness or 799 00:51:16.750 --> 00:51:19.700 I don't know. There are a lot of things that humans can, 800 00:51:19.700 --> 00:51:22.840 can improve and we cannot be replaced by any artificial 801 00:51:22.840 --> 00:51:24.090 intelligence. 802 00:51:24.090 --> 00:51:27.280 Yeah, this is this Bruce said in the healthcare setting, 803 00:51:27.280 --> 00:51:28.580 you know, people are, you know, 804 00:51:28.580 --> 00:51:31.460 particularly radiologists are worried about being replaced 805 00:51:31.460 --> 00:51:36.200 by deep learning. But what we're starting to say is anybody, 806 00:51:36.200 --> 00:51:40.910 any clinician or any PhD advisor who can be replaced by an 807 00:51:40.910 --> 00:51:43.260 AI should be, and those of us, 808 00:51:43.260 --> 00:51:47.870 the rest of us will just do a better job by using the tools. 809 00:51:47.870 --> 00:51:51.150 So, so I don't think there's any, any worry about that. 810 00:51:51.150 --> 00:51:53.700 Of course I'm closer to retirement than a lot of other 811 00:51:53.700 --> 00:51:54.700 people. But. 812 00:51:57.920 --> 00:52:00.740 One of the, I think one of the interesting things that, 813 00:52:00.740 --> 00:52:05.140 that Open AI did with their chat G P T interface is that 814 00:52:05.140 --> 00:52:08.780 it's not trying to do anything by itself, right? 815 00:52:08.780 --> 00:52:12.400 It's, it's only responding to human questions. 816 00:52:12.400 --> 00:52:15.470 So like we do have a chance to conduct, 817 00:52:15.470 --> 00:52:18.300 we have a chance to magnify our time. 818 00:52:18.300 --> 00:52:21.270 Like what used to take me, you know, 819 00:52:21.270 --> 00:52:26.270 three hours to do things that don't make me smarter now I 820 00:52:26.430 --> 00:52:28.850 can do in 25 minutes. 821 00:52:28.850 --> 00:52:32.230 And that's a, that's a welcome thing. 822 00:52:32.230 --> 00:52:35.590 It sort of changes how I think about certain tasks that 823 00:52:35.590 --> 00:52:37.150 before I'd have to do, 824 00:52:37.150 --> 00:52:40.070 but I'd have a mental tantrum before I actually got around 825 00:52:40.070 --> 00:52:41.330 to doing them. 826 00:52:41.330 --> 00:52:45.400 And now it's, it feels nice. 827 00:52:45.400 --> 00:52:47.500 It, it feels, it feels different. 828 00:52:47.500 --> 00:52:50.190 I think, I think there probably will be, you know, 829 00:52:50.190 --> 00:52:53.930 you could spin up a research assistant who, 830 00:52:53.930 --> 00:52:58.930 who is an AI and train it so that it, you know, 831 00:52:59.200 --> 00:53:02.760 is specifically trained for the way you think about things 832 00:53:02.760 --> 00:53:06.470 or maybe in ways you haven't thought about things to help 833 00:53:06.470 --> 00:53:11.470 synthesize, we're experimenting with, with data analysis, 834 00:53:12.000 --> 00:53:14.760 qualitative data analysis as well. 835 00:53:14.760 --> 00:53:17.927 I'm gonna be quiet so others can talk. 836 00:53:20.190 --> 00:53:23.760 Well I'll just mention here in, in Utah we've been, 837 00:53:23.760 --> 00:53:26.910 some of my colleagues have been playing with chatbots for 838 00:53:26.910 --> 00:53:31.910 some clinical things like, oh, for a while that we were, 839 00:53:32.000 --> 00:53:33.550 they were working with, 840 00:53:33.550 --> 00:53:38.550 with COVID testing and covid vaccines and then advice about 841 00:53:38.840 --> 00:53:41.720 lung cancer screening and so forth. And those of, of all, 842 00:53:41.720 --> 00:53:45.720 of course just been rule-based algorithmic things. 843 00:53:45.720 --> 00:53:49.400 And it'd be interesting to ex extend that to something like 844 00:53:49.400 --> 00:53:50.370 this. 845 00:53:50.370 --> 00:53:54.120 It it's, it's nice because the, the accessibility of the, 846 00:53:54.120 --> 00:53:55.880 of the chat thing, most, 847 00:53:55.880 --> 00:53:59.280 most people with cell phone can have access to that and they 848 00:53:59.280 --> 00:54:00.960 don't all, all have, you know, 849 00:54:00.960 --> 00:54:05.000 the super smartphone to do the fancier thing. So, so this, 850 00:54:05.000 --> 00:54:05.840 this could be a, 851 00:54:05.840 --> 00:54:10.700 a real interesting addition to our fairly successful chatbot 852 00:54:10.700 --> 00:54:15.700 communication in, in some simple healthcare areas. 853 00:54:18.710 --> 00:54:20.410 For sure. One of the things that, 854 00:54:20.410 --> 00:54:25.370 that we're working on is getting it in the background to 855 00:54:25.370 --> 00:54:26.970 write the, you know, 856 00:54:26.970 --> 00:54:30.690 the database query to search to see if it, 857 00:54:30.690 --> 00:54:35.690 if what they're talking about is in it, if it's not at it. 858 00:54:35.990 --> 00:54:37.370 And so, 859 00:54:37.370 --> 00:54:41.810 so it can sort of be sort of self growing as it's going on, 860 00:54:41.810 --> 00:54:46.330 especially in the area of math and math learning a lot of, 861 00:54:46.330 --> 00:54:49.140 well you all probably had great experiences in learning 862 00:54:49.140 --> 00:54:53.880 math, but a lot of people have had, you know, 863 00:54:55.200 --> 00:54:57.800 terrible experiences where the person teaching just thought 864 00:54:57.800 --> 00:55:01.883 it was about reproducing with fidelity, you know, 865 00:55:03.210 --> 00:55:08.210 calculations instead of, you know, all the stages that 866 00:55:10.860 --> 00:55:13.910 Conrad Wolfrem would talk about that goes into math. 867 00:55:13.910 --> 00:55:18.910 This is an amazing tool chat, G p t is an amazing tool for, 868 00:55:20.920 --> 00:55:23.670 for thinking together about math. 869 00:55:24.610 --> 00:55:26.460 What, like, what are you interested in? 870 00:55:26.460 --> 00:55:28.620 What do you know about how does that connect to math? 871 00:55:28.620 --> 00:55:31.820 Help me understand these concepts in a way that we didn't 872 00:55:31.820 --> 00:55:33.280 have ACC access to before. 873 00:55:33.280 --> 00:55:35.980 And it's doing some things with markup now too, 874 00:55:35.980 --> 00:55:40.980 where it can give you sort of visualizations in that, 875 00:55:41.390 --> 00:55:43.630 in that direction. 876 00:55:43.630 --> 00:55:48.630 So, you know, I I I think it's gonna, until it kills us all, 877 00:55:49.810 --> 00:55:54.690 it, it's gonna magnify what we can do and maybe the, 878 00:55:54.690 --> 00:55:58.023 the PLE ability with which we can do it. 879 00:56:01.060 --> 00:56:03.870 Well that's a, if that's not a great line to close on. 880 00:56:03.870 --> 00:56:08.030 I can't think of think can't think of one until it kills us 881 00:56:08.030 --> 00:56:09.630 all. 882 00:56:09.630 --> 00:56:10.420 That would be another. 883 00:56:10.420 --> 00:56:12.620 Only the bad professor. 884 00:56:12.620 --> 00:56:14.590 Alex Freeman. Yes. 885 00:56:14.590 --> 00:56:18.490 Another Lex Freeman podcast by El Aer, 886 00:56:18.490 --> 00:56:20.790 I'm blanking on his last name. 887 00:56:20.790 --> 00:56:22.660 He sort of tries to balance, 888 00:56:22.660 --> 00:56:25.480 have the CEO on and then have someone who's like, Hey, 889 00:56:25.480 --> 00:56:27.960 we need to work on the human alignment problem and the 890 00:56:27.960 --> 00:56:29.530 verification problem. 891 00:56:29.530 --> 00:56:34.070 So wherever, wherever you sit it, it's, 892 00:56:34.070 --> 00:56:38.280 it's been a really exciting last six months, couple years, 893 00:56:38.280 --> 00:56:39.780 looks to continue. 894 00:56:42.040 --> 00:56:46.820 Well that is all the time that we have, so I, 895 00:56:46.820 --> 00:56:50.050 on behalf of everybody in the center, 896 00:56:50.050 --> 00:56:53.090 thank you very much for taking the time to present to us. 897 00:56:53.090 --> 00:56:55.270 It has been very enlightening. 898 00:56:55.270 --> 00:56:57.190 Thank you. Thanks for giving us the opportunity. 899 00:56:57.190 --> 00:56:59.950 This is the first time we talked to any group about, 900 00:56:59.950 --> 00:57:03.190 about what we're doing, so we appreciate it. 901 00:57:03.190 --> 00:57:04.230 Thank you. 902 00:57:05.220 --> 00:57:06.430 Yeah. Terrific work. 903 00:57:06.430 --> 00:57:07.750 Thanks everybody. 904 00:57:07.750 --> 00:57:08.070 Thank you.