IGF 2020 - MAG - Virtual Meeting - VI

>> CHENGETAI MASANGO:  Good evening, good afternoon and good morning, everybody, this is virtual MAG meeting number 6, and just the usual things before we start.  This meeting is being recorded and transcribed, I think.  We will be using the speaking queue and the link is being put in chat at this moment.  And also before we start, I see there is a number of telephone call‑ins, can we just identify the telephone call‑ins only, because we know everybody else's name.  So let me start quickly with the person's whose number is 1646, if they could quickly identify themselves.

>> VENI MARKOVSKI:  This is Veni.

>> CHENGETAI MASANGO:  How are you doing?  We have call in user number one.  I have just unmuted you.  If you could just identify yourself.  I think you are the only other person who is calling in. 

>> CAPTIONER:  This is Becky with Caption First.

>> CHENGETAI MASANGO:  So with that, let me give the floor to Anriette to start the meeting. 

>> CHAIR;  Whatever season it is, it's currently the end of summer so we are beginning to get cold weather, and I hope those of you in the northern hemisphere are enjoying some Spring and that your house arrest allows you to enjoy some of the Spring.  And those in the tropics are very lucky.  Our agenda for tonight ‑‑ I'm sorry, just muting here. 

Welcome from the Chair which we had.  Updates from the Secretariat, and we are going to look specifically for the session proposals and received, as you all know, the deadline, and any other news and updates we have got from the Secretariat.

Then, and this is the most important part of tonight's meeting, is to walk through the online workshop evaluation system.  So we are ready to start working on that next week.  And then finally, we will look again for plans for the June open consultation and MAG meeting, and then if there are any other matters that anyone wants to table, just put it in the chat, you can raise it later on.  So on that, I declare the meeting open.  Welcome, everyone, for joining us.  It looks like we have got good attendance, and Chengetai can I hand it over to you.

>> CHENGETAI MASANGO:  Yes, thank you very much, Anriette.

We did have one regret from Rose who cannot join the meeting.

>> ANRIETTE ESTERHUYSEN:  I forgot to mention that.

>> CHENGETAI MASANGO:  Yes, so that's rose.  So for updates from the Secretariat, as it stands for our various calls we have ten Open Forums, 14 preevents, 44 booth requests that have been submitted.  21 expressions of interest for our music night.  We have six hub that's have been registered and 157 travel requests.  So that's, and we have 174 that have been started.

At the moment, we have three workshop proposals that have been submitted.  Now, this is compared to last year eight days before the deadline because the deadline is the 22^nd of April, we had 5 that had been submitted.  So and also we have 27 that started last year with eight days to go and this year we have 28 which is one more than last year.  So basically, as I said during the last meeting, the last virtual meeting.  Everything happens within the last 72 hours.

That's when people start submitting.  They do not use the system to write up their proposals.  They might write it in a Word document and then shift everything compared to last year we are still seeing the same figures, but that doesn't mean we should just sit back and relax.  We really do have to encourage people to submit, especially this time when, you know, people may be feeling a little bit uncertain about the future, et cetera.  So please do encourage people to submit and help them to submit if they need help. 

The Secretariat is here as well, so you can direct them to the Secretariat, and we can help point people out to the resource persons list.  If they are having problems finding, and please do remind them that this year we are making extra effort for remote panelists as well, so if they are not sure whether or not they will be able to make it for the November meeting, it doesn't matter.  There are facilities for remote panelists so please do encourage everybody who wants to and who you think should submit a workshop proposal to submit a workshop proposal.

Another thing I want to highlight is for the hub.  I think the hubs are very, very important where groups of people can gather.  They can gather at universities, at workplaces, at home, wherever, and come into the IGF meeting and participate as a group, as a remote hub.  I think that is also very important, and the earlier they register, the better we can prepare them, the better we can integrate them into the program.  And Luis is there, and he is very willing to give them tips and take them through how they can become very effective remote participants.

Now, as for anything else, I don't have anything else on my list, you about I will ask anyone else from the secretariat, I will ask Anja and Luis, and Lima is on, maybe she wants to update anything.

>> LUIS BOBO:  Nothing from my side.

>> CHENGETAI MASANGO:  Anja.

>> Nothing to add, thank you.

>> Lima?

>> Nothing from my side.

>> CHENGETAI MASANGO:  Back to you, Anriette.

>> CHAIR:  Maybe just a few things, and that is that Secretariat has been working on drafting a newsletter and I have got a message, we will be distributing it even before the completion of the newsletter.  It just has to be logged and the Secretariat has been collecting updates from all of the BPFs and DCs and so we will be sharing that.  I don't know if you have anything to share, but Secretariat, any updates on the translation, on the voluntary translation of IGF messages that see is wed we would do and we open an invitation to host countries to do that.  Has anyone come forward?

>> CHENGETAI MASANGO:  We did open invitation and we did have an email that people could send office to help and Lewis is in charge of that.  I will ask Luis to come and tell us if anybody has responded.

>> LUIS BOBO:  Thanks, Chengetai.  We have a couple of people that wanted to help, but.  What we have done is we have put in the calls.  This is one of the things that was said to be translated the calls for translation.  However, we have not received additional support from that, apart from the interest.  Then there has been, this is by Carlos and the translation fund document, and we are going to do that immediately.  I don't have any other news specifically about the messages, everything is just a document that is translated.  But the system to translate is still open, and we have put content there for the volunteers to translate.  So all of the calls, for example, are open for translation to the UN languages.

>> CHAIR:  Thanks for that, Luis, I think we need to fast track translation of the 2019 output, otherwise it will just lose relevance so let's work on that in the coming week.  And Secretariat, there was one more question I had ‑‑ I can't remember, oh, it will come back to me later on.  There was something else I was going to ask you.  I know exactly what it was.  Could you just update MAG members on the call we had last week.  The outreach webinar which was to my knowledge extremely well attended based on what I could see just if you can give MAG members a short update on that call.

>> So last week from the 6^th of April, we had our webinar which was a global webinar open to everybody.  This follows the one that we did for our Polish stakeholders.  On this webinar we had more than 280 people that joined the webinar, and also Vint Cerf was there so he joined the webinar, and we just told the participants what we were doing for the 2020, the intersessional activities, the themes, the Best Practice Forums, and what we have planned so far for the IGF 2020 meeting including the parliamentary stream, the youth stream, et cetera.

It was well attended.  We had quite a few questions, questions ranging from Visa issues to are we going to have main sessions on the COVID‑19 or the effect of the COVID‑19 pandemic, and the use of ICTs and also from an Internet Governance point of view.

So that was very well attended.  The slides that we used are on the website, so if somebody wants to go through them, they are there and we will put the link into the chat.

>> CHAIR:  Thanks, Chengetai, and thanks for the MAG members that gave input during the outreach webinar and from Shamek and others from the team.  Secretariat you did an excellent proposal so that slide show is there if anyone wants to see it which provides an overview on the themes and the intersessional work and on the location, then please do.

I think my one take away from that is that there is quite a lot of uncertainty out there as a result of the pandemic.  So I don't think we should underestimate that.  Even though it is normal that we have very few session proposals completed eight days before the deadline, I think there are people out there who are not sure that the IGF will take place or whether it will take place virtually.

So it's very important that we do send a message that there will be an IGF, and what we did do during the outreach call was to make a commitment to the community that we will keep them briefed on a regular basis as to any plans, any implications for the IGF as a result of the pandemic.  But thanks very much, everyone, who participated in that, and who prepared for it.

So on that note, let's start with item 3, the walk through the online workshop evaluation system Chengetai should I hand back to you at this point.  I don't see any hands in the queue.

>> CHENGETAI MASANGO:  Yes.  I will give the floor to Luis.  He did a very good job building the system, and with, of course, the input from the Working Group on evaluations and the MAG Chair.  So Luis.

>> LUIS BOBO:  Thank you, Chengetai.  So I'm going to try to be very slow and clear and please, if you have any questions at the end, don't hesitate to ask here or later during the whole evaluation process, I would be ready to help you.  So basically what I'm going to show you is like the system which is based on the system last year incorporating the changes decided by the Working Group.  So I'm going to share my screen.  You should see these from the evaluation system.

This is the link it will be in your dashboard now every time you access our website, you go to a dashboard which gives you the most important links.  One of the links is the evaluation form and, of course, this will be shared when the evaluation starts.

So I'm going to log in with my mock MAG member user.  Okay.  So I will go to the dashboard.  I have the link here, but if you access directly, you will go directly.

Basically as we have discussed in a couple of meetings before, here what we have for the moment is the style of last year with the access to proposals of last year because we have proposals this year yet that can be shown at this stage.  So basically what you see here is the first line is the deadline and the countdown with the time.  And then here when you access, depending on the grouping which you have been assigned which the thematic track, you will be here and you will access to the list of proposals that you need to evaluate.

So here are all of the proposals, the selected proposals of 2019.  And when the evaluation is open here in this screen you will see workshop proposals 2020 and only imagine that you are in the thematic and there have been 100 proposals, you will see 100 proposals.  Then if you click on them you will be able to see the proposal.  This is a proposal of last year, of course, but you will see the proposal of this year with information.

Important to note is that MAG members when you access the proposal, you will have slightly additional information which is the gender, nationality of the speakers.  All do not see the organizers.  Here you are seeing the organizers because this is from last year, but when you open the 2020 you will not see the organizers we are hiding them for clean evaluation.  If there is a conflict of interest you can mark that but we are trying to hide the organisation, so you will see the content of the speakers, of course, for example, this one.  Here is the biography, name, this one doesn't have a picture.

Human rights, the specialty and information here.  Okay?  So you will have all of the proposals here and you will have all of these proposals in a PDF that will be put here.  So last year we had this full PDF with all of the proposals which is the same information you have online but in a unique document, one proposal after the other.

Then we can put an index which you can access directly a proposal and you can read them in the same way that you do in the website, but you can work offline.  Then in order to evaluate the proposal, it's very, very easy.  Basically you will have here a dropdown menu with all proposals you will have to evaluate.  So it's exactly the same as the list here.  You have read this proposal, 180, you would go here to the proposal and you would evaluate it.

Here for this year and for now, I have created two mock‑up proposals for you to practice.  It would be exactly the same, so imagine that this is one of the proposals that you have read here or in the PDF document and then you would select this proposal as the one you are going to evaluate. 

So there are two questions, first is you have or not a conflict of interest.  So this paragraph has been added this year according to the Working Group.  You can decide if it was a conflict of interest or you have a conflict of interest.  Here there are also cases that explain it in which you should create a conflict of interest.  This has been decided by the Working Group.  Members are aware the proposals submitted by the organisation they work for.  Any other reason you should claim the proposal, in that case you would click here I have conflict of interest and then here with a short line with a reason you would click submit and basically it will disappear from the menu of the proposal that you have to evaluate.  It will go right to the schedule with a conflict of interest.

Again, the Secretariat, not all of the proposals are evaluated by all of the MAG members at the end of the evaluation, but we have to have minimum number, which is the same as last year, which is around 7, 8 MAG members should be okay because we have had these cases in previous years and there are more MAG members that elect proposals.  If we have to assign any proposal to other MAG member because there are not enough, because imagine that four, five MAG members declare a conflict of interest, we could assign enough other MAG members to kindly evaluate the proposal or trying to keep the balance in gender, stakeholder group, original group, et cetera.

So this is the first question, or you have a conflict of interest.  Or you don't have a conflict of interest.  If you don't have a conflict of interest, then you would evaluate the proposals in the six criteria that are declared this the review process.  This is the same as last year except from diversity which this year has been defined another way in which the proposals rather would select what were the focuses of the diversity points in the proposal and how they elicit diversity.  So this is what has been explained in the paragraph and put here by the Working Group.

And in any case, what you have to do is try to select your reason of how the proposal responds to any of these criteria by also understanding what the Secretariat, so this is the relevance, et cetera, and how the proposal responds to that.  This has been the work from the Working Group from the previous two, three years, and it's basically for the MAG to try to have the best point of view on what is two points one, two, three, four, five, who can say, but it's needs improvement so for everyone more or less have the same vision and you decide whether it's poor or good.

But, for example, if I put here, I put needs improvement, excellent here, you will see that the resulting score is calculated there at the end.  And same for this case.  You can see that there is weight in the criteria and this is the third and last change by the Working Group on evaluation proposal.  For example, the content has a bit more weight than the interaction.

If I exchange here from excellent to good, it should go 0.2 points, you see, 3.5, however, if I go here on the same one, one step equals 0.1.  It has a little bit less weight.  So it's basically to mark the proposals in these six criteria and then you have three spaces for comments and that's all.  The three spaces for comments are one for the MAG.  This is important because this is going to be shared with the MAG during the June MAG meeting and it will be visible by all.  So at the end imagine every proposal has been evaluated by ten people.  It will have the addition of the ten comments that these people have put in the proposal.  So the MAG will have the average score of the proposal and information the evaluators, what they have said about the proposal.  This can be commented in the June MAG meeting.  The second comment is feedback for proposal organizer.

This is the comment that you want to send to the organizer to the proposal after your evaluation.  Regardless if it is selected or not, they can receive the feedback, and, again, if there are ten MAG members that evaluate the proposal, they will receive the different feedback from the proposals.  Of course, there can be differences but at the end what we have from the proposals is that they get everything as a set of feedback information from the different evaluators, and it helps them to improve the proposal for next year or maybe for the merge or maybe, okay, this is the feedback from the MAG.  And this is the only non‑mandatory field which is internal comments.  This is for you.  This is only visible by you and the Secretariat, everything is visible by us, but this is just to keep track of whatever.

For example, if you make an economic evaluation and you can see at a later state.  So maybe you want to put here a note just to review or just to recheck about the interaction organizer just leave it blank.  So this let's just try to submit, for example, it tells me I need to put something here.  If I put some comment here, I will just put some and feedback to the scoring users, well done, and internal comments not applicable.

If I click on submit, so to evaluation for this proposal is sent and now in my platform, I see again the same as before, the deadline, the access to the list of proposals, but now the proposal has disappeared so it's not pending anymore.  And it appears here.  It appears in two places, one here and one here.  It's exactly the same, just the title, what have in my marks in the final score and the comments.  The point is that this is a table view, but the comments can be a little bit narrow here, and here is like text view which is exactly the same, here are the marks and this can be several paragraphs.  For some of you want to put longer information.  So it's exactly the same.  You can change or update any of the evaluations clicking here and you go to the same place, and you can also continue doing a proposal.  Let's do the second one and you will see how it appears.  So this is send to conflicts of interest, and then it disappeared, so I will mark it.  This one is going to be excellent.  So this is 3.4, I will invent something, here, here, here, I click on this one.  There are no more proposals for me to evaluate and here are my evaluations.

They are ordered by title but, for example, you can order by policy mark and it's going to order from the top to the bottom.  I can also order on this side.  I can order by format here, for example.  And in ascending order, for example.

This one has 340 and 370.  I can sync this one, for example.  I would have, I cannot change the title, I can just change the marks and the comments.  And change this to good, this one to frequently.  I click on save and it's changed here.  So you want it here now.  And it has also been changed here.  Basically that's that.  So basically you can work offline, see the list of proposals, decide at any time which one are you going to evaluate, and so just go to the link, of course, I can log out and log in and I will find my evaluations as they were before.  I can change whatever at the same time.  I can make anyone appear here and the deadliness to complete this will be closed and all evaluations that we have will be integrated.  Basically we take all of them and we generate the upgrades for each proposal.  We make the rankings by theme.  We calculate the standard deviation, et cetera.

You will have a time frame when you can access the evaluation forms.  You will have access to the proposals offline and online and you can start to put the evaluations.  And that's all from my side.

>> CHAIR:  Thank you very much, Luis, and the process has become so much easier since I was on the MAG.  I found a chat, it was one question from Karin about the maximum score per category, and if I understand correctly it's all excellent, is that correct that the maximum score is always five.

>> LUIS BOBO:  That is completely correct.  They are weighted, but you put 5, 5, 5, 5, 5, the weighted average is still 5.  If you put 3, 3, 3, 3, it would be a 3, of course.  It's 1 to 5 to maximum is 5.

>> CHAIR:  So it might be just maybe make a note of that at the top of the form just to make it visible, although I think people will pick that up very quickly.  Any other questions?  I don't see hands.  Speaking queue is empty and I don't see hands.

>> CHENGETAI MASANGO:  There was a question about having subset of the work proposals to evaluate.

>> CHAIR:  Why don't you take the mic and ask.

>> Hi, can you hear me.  Just a quick question following, is it possible to get PDF of our assessment in addition to the full set of, all of the proposal, and second is the comment on the organizers.  The names of organizers should be hidden but gender, stakeholder, and relational groups should be, and, this was last year, should be shown so that we can actually use those as elements for evaluation of diversity.

>> LUIS BOBO:  It is for you and information on diversity, so you can see if the first organizer is from what country, what gender, what stakeholder and what regional group, you just don't see the organisation and the name, of course.

That's one thing.  And the second thing, here is the general list of all of the proposals and here will be, so last year it was security, safety, stability, resilience, et cetera.  This year it will be, trust and data and environment and inclusion.  Every MAG member, so these groups are already being created at the moment.  So these are balanced groups of evaluators, each group will evaluate one of the thematic tracks and, of course, we will know which group it will be.  So the Working Group on evaluation proposals is that the facilitators, the heads of these groups will be in the groups, but then we have to build random and well balanced groups of MAG members to evaluate the different tracks so you will see in what group you are and you will access the PDF of those proposals only.

>> Just one more.  As a member, if I check my proposal, the proposals are from different thematic groups, not from the same thematic group.  They were randomly assigned to each of the MAG members so each MAG member has approximately the same number of proposals family.  What can happen if you use this system it can have, for example, data governance, just imagine, 50 proposals and digital inclusion, we can have 150 proposals.

So the group, the MAG member that gets digital inclusion to evaluate will be three times the burden of a MAG member that evaluates data governance.  Last year we used the system that all proposals were randomly assigned to each MAG member.  We did not, we did not use the system to assign MAG member to certain, to certain group.

>> LUIS BOBO:  Sorry to correct that thing, but last year we did use thematic tracks.  The point is that we also thought on the possibility of receiving many proposals of one track and less from other tracks.  The point is finally that is what the number was really well balanced.  It was something like 92, 103 and 109, something like that.  So the MAG indeed explicitly agreed, it doesn't matter if someone of us evaluate 93, others 108.  Also.  Some of us can have conflict of interest and reduce.  That was what happened last year.

In previous year it was also thematic although the only difference was that the MAG members could select things of interest.  I think this was waived last year, and it has been waived again this year according to the Working Group on evaluation of proposals.  And then in case that this would happen, so let's see what happens, only after the proposals have been submitted we can know that.  The solution is like, there are 50 MAG members so if one of the groups imagine there are 100 proposals, 100, 100 and 200, instead of four groups we will have five groups, and the one with 200 proposals will be splitting the most possible well balanced groups to evaluate each one of them 100 proposals of the same track and if there are two facilitators they will be one group.  To we only receive that when we receive the last number of proposals, we can decide.

>> CHAIR:  Thanks Luis, Nabosa, is that clear, so there will be randomly assigned groups according to theme, but the Secretariat will make sure that all MAG members have more or less the same working load.

>> I understand, and that was mentioned last meeting that they had online.  I will once again check my proposal that I forgot here and for me it would be better system if we have, if we have proposal randomly assigned from all different groupings rather than one particular group, so but it's up to the core MAG to decide on that.

Thank you very much, and thank you to Secretariat for making this, I think it's a good system that we used last year and I think it will continue.  Thank you.

>> CHAIR:  Thank you, Nabosa, and thanks very much to the Secretariat and thanks very much to the Working Group for working so, not just working hard on this, but pushing a lot of thought.

>> It, doing the survey and improving process based on the input from participants from 2019.  Does nick from the Working Group want to explain your thinking behind the way in which MAG members will be divided to do the evaluation.  I see quite a few are online.  I think Nabosa's question is important, so if somebody can respond to give us a sense of what your rationale for this way of approaching it.

>> ROBERTO ZAMBRANA:  Yes, if I may, Madame Chair, this is Roberto.

We went through different discussion about how to, we were supposed to allocate what could be the best approach, and among the different alternatives including one that reflected what we did, I mean, the MAG decided to do in the previous years, finally we agreed in this intermediate approach, and this means that not all members were supposed to be assigned randomly, but leaving at least the members that were already involved in the tracks regarding the narrative task that we had, and after, after that all of the other members will be assigned differently.

The other, the other thing that was in discussion was if we actually could have a preference regarding which group would someone like to be, but, again, most of the group said that it should be better if we provide this kind of balance considering that everyone of us is capable to participate in any of the four groups, and that's why we finally decided that way.

I hope if any other of our members of the group could say something that would be good too.  That should be from my side.

>> CHAIR:  Thank you very much, Roberto.  Does anyone else want to add?

>> BEN WALLIS:  Anriette, it's Ben Wallis.

>> CHAIR:  Go ahead, Ben.  

>> BEN WALLIS:  I can add to that and add to other comments I was going to make about how to tackle a workshop evaluation.  So in terms of why we would break it up so that MAG members are focusing on one in particular thematic track, the idea was to give the better sense when you are marking different workshops of the similarities of the topics that are going to come up in that track, and so it's easy for you to compare and get a sense of when one is stronger than the other and to allow you to go back, you know, in the trust tracking it might be the ones in cybersecurity, as you mark more and more that are related to cybersecurity, you might get a better sense of actually that one is quite strong and maybe I marked this one too highly and you can go back and adjust.  Because you are looking at workshops which are broadly in a similar area.  If you were assigned workshops across all thematic tracks on various topics, you wouldn't get the ability to compare and understand what people are looking at on different topics.

That's valuable not just when you are doing a workshop evaluation, but when we get to the next stage afterward, when we try to balance out the successful workshops to make sure different topics are covered and potentially to look at what might be worth merging together.  That's easier to do if you are looking at one track rather than your 100 issues being from all across the gamut of issues.

So that's what I would add on why we are assigning MAG members workshops on a specific thematic track.  I did want to say that people will have different whys of working, and I'm one of those people who likes, who can work offline rather than working in the platform.  So for that reason it's incredibly handy to have the PDFs that the Secretariat produced last year and I see on the chat they will provide this year.  That's helpful to enable people who want to work offline.

Some people might want to do it in a Word document, some people might prefer Excel, that kind of thing.  I the last two years have produced a Word document which I kind of fill in as I go and I can share that once I produce my little template with MAG members in case anybody want toes to borrow that way of working but people will have different approaches that work for them.  The final thing I would say is about the conflict of interest, and this has been said in previous meetings, but it's really important as a courtesy to your fellow MAG members indeed that in the first few days once you receive the workshop and proposals that you go through and identify any which will be a conflict of interest and you go into the system and mark them as conflicts of interest because that allows them to be reassigned to other MAG members early in the process.  We have been given four weeks to do the evaluation this year, which is generous, it's a week longer than the last two years and it should be enough time, but it is a time‑consuming exercise and you do need to allocate yourself plenty of time during the four weeks and not leave it to the last few days.

If people leave the exercise, if they don't identify conflicts of interest in some of the last few days then those get put back into the system and MAG members are being given new ones at quite a later stage.  I'm sure it would be worth the Secretariat reminding people once we will receive the workshop proposals to evaluate that that would be a great first step for everybody to take in the first few days.  Thank you.

>> CHAIR:  Thank you, Ben, that's a very important reminder that the conflict of interest should be indicated early on.  We have in the speaking call, Paul and then Jennifer and Jutta.  I wanted to ask the old members, not old in years but MAG members who have done this before, when you take the floor, just share, and I'm inviting other MAG members as well, if you can just share some tips with new MAG members and something that you learned from making a mistake or something that you felt is good advice that has worked well for you.

So when you do take the floor, just share if you have such a tip with new MAG members.  Paul, you are next.

>> PAUL ROWNEY:  Thank you.  Paul Rowney for the record.  On the last note, it's time consuming, exhausting process.  I don't think there is an easy way to navigate through it.  It's just a matter of, as they say here, eating an elephant one bite at a time.  It's a process of just, we have to work through.

It doesn't get easier.  It starts, it ends.  From my perspective, I try to target a certain number per day.  It's good in my opinion not to try and work through all of the workshops too quickly.  You need time between working through the different stages of the different work shots that you are plowing through.  I also wanted to support Ben that ‑‑ I support the workshops being assigned by track to groups of people so you are focused on workshops in a specific track.  Not only does it help you to evaluate the workshops against the other workshops you get a sense of the breadth of the different workshops and repetitive workshops.  It's possible to go back and revisit some of the work shots, but it also helped with the post evaluation process where we need to look at balancing the thematic track.

  I just have some comments that I want to add and I apologize I haven't posted this to the Working Group.  The last month or so has been hectic for many different reasons, but we still have these five groupings of how we evaluate each of the questions, needs improvement, good, excellent, and from my experience, it adds a lot of work thinking through.

It's easy to say that policy questions are absent, but on some of them, being absent or needs improvement, they are quite close to each other.  I'm wondering whether we actually need five bands.  Absent and poor are pretty much the same or poor, needs improvement are pretty much the same.  Good and excellent I think are good.  I don't know where we can rationalize them down dough four different categories.  It would certainly make the evaluation process easier.

You spend a lot of time thinking is this poor or does it need improvement, you know.  Rather than saying, okay, they are not there, they don't exist or it's poor but needs improvement or it's good or excellent.  It's a lot easier, and I think that would make the process a lot quicker to evaluate, but I know it's a little late in the process.  And from experience I also want to comment, we have this compulsory feedback, it's called the proposer organisation.

The first year that I did this, this was an option.  The Secretariat year it was compulsory.  At the end of each evaluation we have to put some feedback for internal use which is compulsory, and then we have to put some feedback for the proposal organizer and that feedback is shared with the organizers, and it's not filtered, and sometimes you are just putting feedback for the sake of putting feedback because there is not really anything to say, so we have to put something otherwise we can't proceed to the next question.

So those are some things I wanted to share.  Thank you.

>> CHAIR:  Thanks for that, wall, and I get a sense ‑‑ Paul, and I get a sense that Working Group members will respond to your proposal about changing the scoring.  Thanks a lot for sharing all of that.  Jennifer, you are next.

>> JENNIFER CHUNG:  Thank you Anriette.  This is Jennifer speaking.  I actually wanted to respond to one of the last points that Paul just made regarding the comments that are compulsory that we have to give feedback.  I must have been one of the people last year that really supported this because I think at least this is my personal opinion that if proposers spend a lot of time and effort and energy to come up with a proposal, we also as MAG members have the duty to, you know, give them feedback on especially ones that may need improvement to give them pointers and tips and at least from what I have heard in my community, especially from Asia‑Pacific, they found it really useful to see ways and comments and that they can improve writing the proposal.

Especially for people who don't have English as a first language, it's really useful.  So this is one of the points I wanted to make sure was also in this year, because when Luis was going through the walk through just now, which was very, very comprehensive, I wasn't able to see if that was compulsory or not, so I'm glad it still is.

The second thing I wanted to mention, and this is more of a question, is last year when we did the three thematic tracks each track kind of worked together ‑‑ first we did separate evaluations, but then each track kind of had its own organic way of narrowing it down or discussing between themselves how then to go forward with the workshops that they want to present to the full MAG.  I am not sure if this year we have something that's a little more structured.  I'm not sure if this year we want to have something more structured, but the reason why I'm bringing this up is each track last year, we had three tracks last year.  We have five tracks this year.

Each track had different ways of thinking about merging.  I don't know if we want to have it a little more standard across the board for if we want to continue to have it kind of facilitated by the people who are leading each track.  I don't mind really either way, but it would be nice to have guidance on that.  And lastly, just to share some tips from having done the evaluations for the past two years, I find that we definitely use the comments that you can write for yourself.  I found that extremely helpful when I was going back through my 100 or so scoring just to see whether or not I have been harsh or lenient.

I also recommend what Paul said, it is not an easy process.  Try not to split it over too many days because you would find that each day you may or may not be, you know, in the frame of mind to continue to look at certain aspects, and you may find yourself scoring higher or scoring lower depending on the day.  And I also find it very good to go back over to see if there are certain workshops or themes that keep on coming up that can be looked into as a cross cutting kind of theme for the entire track, or if it's possible to do some merging.

I have been using that particular comment box just to make notes for myself.  Sometimes I also put it in the one where it is shared with the entire MAG so people can see, you know, certain proposals would probably be beneficial to be merged together.  So those are my two cents, but mainly my question is I don't know if we want to have a more structured way of going about this.

>> CHAIR:  Thank you, Jennifer.  Jutta, you are next and possibly if you can also respond to Jennifer's question, that would be good.

>> JUTTA CROLL:  Okay.  Thank you Anriette for giving me the floor, and I want to apologize first for interrupting Paul with my comment.

I thought I was muted, but I really thought oh, no, to a suggestion of reducing the scores from five to four.  I will explain a little bit later that issue.  First of all, I wanted to say that all of the efforts that went into this system over the last year were based on the objective that we want to create a program that is based on the best proposals that we can get.  So we need a neutral assessment when we do the job in assessing all of the workshop proposals.

We need to be neutral.  We need to be as objective as possible, and with a system that is much more easier than with having only you're own impression of a proposal and trying to put that in words or scores or whatsoever.  So I do think that's the system now is really helpful, and to be honest, I also do think that there are differences between the five scores that we now have, and absent of something else, then poor and poor is less good than needs improvement.  You could explain all of these five scores with more words and then it might be more better understandable.  So needs improvement needs it's not too bad, but it could be better.

So you have a scale of these five scores and as Sylvia already wrote in the chat, we might review that the last, in the next year, but this year I do think it's just too late to change that.  And from previous processes of assessment from other competitions also from assessing proposals for Articles in journals and scientific journals and so on, if you do that there is a four score scale, you get not as good results as if you have a five score scale.  So it's really better to work with five different scores overall.

And my last point would be, Anriette, you have asked for tips for those who are doing the job of workshop evaluation, proposal evaluation for the first time.  I do think it's, of course, you know it will be a burden.  It takes a lot of time, but just give it a start.  Try with the first two or three or four proposals.  Try it out, see how it feels when you see the overall score, and then maybe you need to go back to some of the proposals, but don't be shy.  Just start it, give your scores for all of the criteria, see how the overall result is, and then you go to the next and then to the next.

You can be sure that you will get quicker and quicker.  You will learn on the process and in the end, it's really not a so difficult task as it might seem now that we have had all of these discussions.

>> CHAIR:  Thank you, Jutta.  Do you or anyone else from the Working Group want to respond to Jennifer's question about standardizing or becoming more standard in how the different groups approach decisions around merging?  Is that something that the Working Group discussed in.

>> JUTTA CROLL:  I'm sorry, I forgot to answer that question.  It was just raised in today's call of the Working Group, and we said that there are some thoughts how the approaches of the four or maybe we have this year five groups who are doing the assessment on their share of proposals, how the approach could be, how do you say it, more Reliant to each other, and we came up with the suggestion that we postpone that a little bit, because we were busy and we are busy now in the last days when getting pro proposals but the Working Group has it on the gained to deal with that issue and hopefully the Working Group will come up with a suggestion and then get back to the MAG.  I'm not sure if it is possible to do that in the next MAG meeting or maybe in four weeks' time, but definitely the Working Group will come up with a suggestion for the MAG before we have our where the final decision on workshop proposals shall be taken.

>> CHAIR:  Thanks for that, Jutta.  And maybe there will be other issues that come up such as how to identify sub themes.

Sylvia, do you want to add on this topic before I pass onto Timea and Maria.

>> Thanks, I hope you can hear me.

>> CHAIR:  We can hear you clearly.

>> I wanted to reinforce a couple of comments that I made on the chat to clarify that one of the reasons to work on that text that complains what absent or needs improvement, good and excellent mean, in all of the six criteria was to actually try and help MAG members to assess in a more, in a similar way what the one, two, three, four, five actually mean for each criteria.  So it's not only the numbers and to have the scale that helps to identify the differences among the proposals and all of that, but the meaning behind keep one of those numbers is actually different.  If Luis could go, for example, to any of the material he presented during the presentation of the online system and show the difference.  There are some, for example, that said in the diversity criteria if I remember correctly it's like, okay, it is absent if only one or less criteria, diversity criteria is assessed, something like that.

And then excellent means more than four or five diversity criteria are assessed.  So that will keep the support for MAG members to say, okay, I have in, it's not only your subjectivity to apply the 1‑5 score and the words behind them, but the wording that went with it is important.  So it is really important that the MAG members familiarize themselves with the mock‑up that we prepared.  That is really useful to understand what the differences are.

And after doing a few, it's easier to, you know, your brain just kind of starts adjusting to it.  But it's really important to try to assess that not only a subjective, but try to follow those guidelines at least to the very minimum so that the scoring across the different MAG members actually means the same.

So when a MAG member said this is good, it means it's good for other MAG members.  So it's not, there will be differences on how we see the overall proposal, of course, but not that many different as the criteria level, and that's when the little differences, the variances between one proposal and the other and on specific criteria could help advance a proposal to a next stage because that's so different, those variances are the ones that are going to help with the cuts, let's say, where you draw the line about which proposals are accepted and which ones are not.

It's very difficult to have like a super balanced view of everybody.  Everybody has, everybody comes from different cultures and different stakeholder groups.  And our culture and our experiences help us to do the evaluation, but also generate differences.  So it's important to use those words in my opinion.

I was the one that pushed more harder for this to make sure that we have, that we can explain to each other what we meant You score 3, right.  So those words, please take a look at them.  They are also on the Google document that I shared a long time ago that has how this system came to light and it has the work of several people that actually put their minds into what wording should go behind each one of the numbers.  So it is as Paul said before, it is long and it can be tedious to do it, to have the wording every time, but you will be surprised about how this how your interpretation of 5 could be with the text or the information from a proposal.  So I hope that you can give it a go again.  Thanks, Anriette.

>> CHAIR:  Thank you, Sylvia.  I also just want to urge people to use the notes.  I think, Paul, you were questioning whether it should be compulsory, but I think it can be useful to make notes to yourself because sometimes you forget when you score lots of proposals why you scored one particularly badly or particularly well.

So I think it's quite useful to use those notes and as a self‑correction exercise as well.

>> PAUL ROWNEY:  Anriette, just quickly, there is a section for internal comments, for writing your own comments.

>> CHAIR:  That's right.

>> PAUL ROWNEY:  Already in there.

>> CHAIR:  So there are internal comments and then there are comments to the proposers, that is correct?

>> PAUL ROWNEY:  Supporting comments which goes to the Secretariat, the feedback to the proposer organisation, and then there are internal comments.  Three comment boxes.

>> CHAIR:  And the internal comments are visible to all MAG members, correct?

>> PAUL ROWNEY:  They are private for yourself.

>> CHAIR:  Only to yourself.

>> PAUL ROWNEY:  These are the ones that you put down and say I might go back to this or could be a duplicate or whatever.  That's just for your review when you look at all of the ones you have reviewed.  That's how I use it anyway, I use it to make my notes so I can go back and understand why I'm going back to that particular workshop.

>> CHAIR:  That's what I was suggesting is actually a good, it's a good mechanism to check that one scores in a consistent way.  So next we have Timea.

>> TIMEA SUTO:  Good evening, everyone.  I hope you can hear me right.  I wanted to share a couple of reflections on how I work when I grade these proposals and I have had the pleasure of going through a few of these and I'm looking forward to my last one.  I guess you can guess from this that this is not the easiest thing to do.

It's not the quickest thing to do.  But as Jenny first said, I think we owe it to the community to give it our best effort.  And also to give it our, our views and our feedback to the community because they do ‑‑ it takes a lot of effort and a lot of work to put a proposal together.

And year by year because the program gets so rich with other content, we have more and more competition for slots.  So it's always a little fraction of the workshop proponents do get their proposal through.  So especially for those that you see already from your scoring that are probably will not be among the winners, if you can call them that, we owe it to them to give it a little bit of feedback of why we think so.

We might, of course, think about how to do that better, how workshop proposers receive the feedback from different evaluators.  Maybe they would have some comments on that in the workshop survey, but I think overwhelmingly the feedback was positive to get some information from the MAG members.

That was one of my feelings.  As for Jennifer's comments around this one.  I wanted to support what Jutta and Sylvia said about the grading system.  I do encourage colleagues, especially those who are first year MAG members to go through the background information and the guidelines for evaluation and the documents that Sylvia referenced.  They do represent a lot in getting your mind around what a score is.  And do try and think about it in a, not in a numerical way, if you can, because it does help differentiate between what absent and poor means and what one and two mean.

How I do this and maybe this is helpful to you, I usually copy and paste the titles of my work to evaluate into an Excel sheet that I create.  Basically the table that Luis showed us I create in an Excel sheet and then I go to my PDF one by one and score it that way.  I tend to work offline better, and it's easier not to get lost in the long evaluation form, and then I transfer those scores back into the system.

And this also leads me to a question whether or not there is an easy way for those of us who work offline to import the Excel sheets back in the back office so that we don't have to go through it one by one.  That would be helpful.  Some of my colleagues, I know, work the same way, as we compared notes in past years, and if that would be possible, I think it would be very much appreciated.

And last also to respond to Jennifer's questions, and to complement what you just said about the workshop evaluation Working Group, we will be looking at having a more uniform way of reaching a decision on how to select the final workshop.  In the coming weeks, of course, everybody who has done this in the past year is welcome to join our group and contribute, but I think what would be very useful is those people who facilitated the selection of workshops for the various groups last year who led this process, if they could give us maybe four or five sentences of what was their process inviting so we can take those lessons when we look into finding a system for this year.  I think that would be very helpful.

I think that's everything I wanted to say.  Sorry if I was too long.

>> CHAIR:  Thanks Timea, and Maria Paz you are next.  And if you could respond to the question about an easy way of importing offline scores into the system.  And but now, Maria Paz, I give you the floor.

>> MARIA PAZ:  Thank you.  Good evening, everyone, and just she short comment, mostly trying to provide my personal experience in the process last year.  It may be something that could be useful for the last question that was made about how to improve viewing the selection of the workshops in each track, so those two for me are linked because one of my advice for new MAG members and for anyone that wants to take it, if precisely that maybe something that is very helpful for having an overall sense of the different proposals, but for a specific track and to have a more (?) view in topics that we think should be considered as part of that track is to take a first round of a general look of the whole proposals that are provided to us as part of that track in order to identify, for example, main topic that you can see in the proposal.

So I did that last year, I also used a spreadsheet in which I categorized and I said, for example, these are the proposals for international transfer of data, whatever.  So in that way, after that general overview that I know what is in my bucket, and I already identify things that are relevant from the track that I personally think is my evaluation that should be in that track.  And I have the possibility to assess in a more progressive way proposals that are referring to the same topic.

That is why I was insisting in the topic composition in an earlier stage because I think that it's relevant that when you accept the proposal, you have the ability to compare a proposal that covered the same subject, and that way to score and evaluate them in a strategic way in order that you can either select the best one of the topic that should be in the track or ‑‑ covered in the track, or facilitate the possible merger soar things that you will suggest to organizers to improve.

And I think that that exercise also was useful at least for the track that I participated last year at the end when we have to finally select a proposal, so there were different scoring and at the end we were less sure about the last slot that needs to be covered, but this idea of having another view of different topic that should be covered in the track helps us to identify what is missing and to look at the good proposals in those topics in order to ensure that they will be covered in the program in a specific track.

So that will be my recommendation both for in the evaluation but also something that could be useful for each one of the tracks to do this final assessment whether they have decided which proposal makes the final selection.  And my last piece of recommendation which is in my personal experience is also the thing that Jennifer is the idea that this personal note that you can make to yourself but also to the rest of the group is really relevant.  It also pointed out this relationship between different proposals that you can start to see when you are reviewing.  So, for example, in those notes I put to myself and sometimes to the group the suggestion that maybe this proposal could be merged with another one.

And if you are very organized like the numbers of different proposals, that facilitates a lot if you need to go back in the review and decide or change scoring or decide which one is better or finally decide this is a good case or a merger, whatever, but you have already that information in the page in which you evaluated, and I did it manually.  I work in my spreadsheet like just putting in the last column this information and then transfer it to the online form, but I have that information for myself, and that's helped me a lot a further are you view at the end in order to refine my scoring and ensure that at least I was pointing out giving the best scores to the best proposal in each one of the topics that I considered that are right on for the programmes.

I will stop there.  Thank you.

>> CHAIR:  Thank you Maria Paz.  It sounds it's if we need to consider capturing what emerges as possible sub themes while we score the workshop proposals.  Is that what you are proposing, Maria Paz, that we use the supporting comment field or some other way of recording that?  What I want to suggest is, and you can respond, Maria Paz, but I do want to ask the Working Group, seeing that the Working Group will be coming forward with suggestions around how to best handle merging would you be willing to come up with suggestions on how to handle the process on how to identify sub themes or mapping the themes more effectively.  Is that something you could take on in the coming weeks.

>> CHENGETAI MASANGO:  The Secretariat does produce a heat map of the emerging sub themes.  It's part of the analysis that we do.

>> CHAIR:  That's perfect.  And what input data do you use for that Chengetai.

>> CHENGETAI MASANGO:  Just the topic themselves.

>> CHAIR:  The sub themes or the issues that they select?

>> CHENGETAI MASANGO:  Yes, yes.  I'm sorry, my mic was off.

>> CHAIR:  So there will be some data prepared by the Secretariat to help with that as well.  And I don't see any other hands.  And we have 40 minutes left.  Shall we move onto the next Agenda Item.

>> CHENGETAI MASANGO:  I would like to also highlight the timeline and I put the link into the chat.

>> CHAIR:  In fact, can you open it?  I was going to suggest, Luis, if you can open the timeline for us just so we can visualize the next steps.

>> CHENGETAI MASANGO:  Yes, so just quickly, because the deadline for admission is 22^nd of April.  And then we have reduced the Secretariat's time to four days to do the, you know, clearance of those that fit into the requirements because some don't and that will just make it easier.  And then, and also to assign the work, the workshops to the various MAG groups.

And then the workshop evaluations the MAG can take four weeks from the 29^th of April to the 29^th of May to evaluate the workshops, and always, always, always, people run out of time.  It doesn't matter how long we give them.  It's just the same thing as with the workshops, submitting workshop proposals.  You can forget about the beginning time.  It's always the end time when people do their evaluations, so please, a number of you have said it, I think some have said it, please just make sure that you assign enough time to do the workshop evaluations.  It doesn't matter which method you use.  You can do it all in one go.  You can break it up into threes or you can do it by the day.  Everybody works differently, but let's try and keep to that deadline of 29^th May for the workshop evaluations to be done.  Will.

And then the Secretariat processes the evaluations.  That's for one week from the 1^st to the 5^th of June and then we give the evaluations back to the MAG, we post it on the MAG list and the MAG can digest them, ask questions, you know, discuss in the various groups in time for the second open consultations and MAG meetings.  Thank you, Anriette.  Back to you.

>> CHAIR:  Thanks, Chengetai.

If there are no questions on the timeline, I don't see any hands.  I don't see anyone in the speaking queue.  Shall we move onto the last item or the second to last item, which is the discussion of the timing of the June open consultation and MAG meeting.  Chengetai.

>> CHENGETAI MASANGO:  Thank you, Anriette.

So we as the Secretariat and also with the MAG Chair, we have been just thinking about the June meeting.  As we said, we have changed and we are planning it as online meeting.  If things change three weeks, 21 working days before the meeting, we can see whether or not we can process travel for people to come to Geneva, but looking at how things are, that is very doubtful at the moment.

At best we may have some of the MAG there and some of the MAG not there, but that will also have its issues.  But thinking it as a virtual meeting, we were thinking that we should not be thinking of it as like a linear, you know, face‑to‑face meeting.  It's just a face‑to‑face meeting basically happening online.

I mean, since we are online, we should take advantage of that, and we also should be as considerate as possible to the time zones that people are.  So, you know, some people are on the west coast and some people are in Australia or et cetera.  So we should try and be as considerate as possible and try and encourage as much global participation as possible.  So this is taking a modular approach to it, and looking at the June meeting that we had last year, just as a starting point, last year's June meeting was packed and we said one of the comments that I think there was some general agreement is that it shouldn't be as such a packed meeting especially if it's virtual and we should also try and spread it out.

So for instance, I have just put in the link for the June meeting that we had last year.  I mean, we have certain things that we know are there.  So the document agenda, welcome from the Chair, from UNDESA, and we might have a message from the USG, et cetera.  Now, also going down to the briefing on the state of preparations, and also from the host country as well, these can be either recorded and then people can listen to them or we can do it for the first run, we do it live, and we also record it because their aim is for the first day it's the open consultations.

We can have an open consultation session in the morning for those people in the east, Australia, Asia, et cetera, and then in the afternoon we also have an open consultation for those people in the west, the Americas, et cetera.  Now, basically these are just ideas.  When I stop talking, we can, you know, say yes, no, whatever.

So we have the session so that we know that we have input from people from, you know, both sides of the globe and also, you know, I am just assuming, of course, that Geneva is the centre of the world, of course, and from the centre and those people in the timeline, they can choose whether they want to attend the morning session or in the afternoon session, but we won't have a repeat of those.  We either have two recordings or one recording.

And also during the day, once we have got the recording for the morning, we can have people submit questions for those sessions, and then in the afternoon session or even during the day, those questions can be answered by those people that have got questions of the host country, then in the evening, the host country can answer those questions or if somebody has questions for the Chair, et cetera.

I sense you get the point on that one.  So we can have it that way, we can have it modular, we can have recordings, online questions and live questions as well at the end of the day.  For the MAG meeting, last year we did have a lot of presentations.  We can also keep those.  We had the report on the, from the MAG or from the digital inclusion evaluation group, so we can have, you know exactly the same thing, report of the MAG from the trustee evaluation group, et cetera.  These as well don't need to be live.

They can be or it can be a mixture of live and recorded.  I'm sorry for picking on Ben in case Ben wants to say something, he doesn't have to wake up at 3:00 a.m. in the morning for the morning session or we can have the first session the day before in the evening and then have that recording for, you know, for what Ben is going to say in the morning.  And we have some live as well for those people who are up and about in the morning segment.

And also we are going to be breaking up into groups.  So for the breaking up into groups, they don't have to happen concurrently as well.  It's depending on the makeup of the groups which Luis assigns you all in.  You can decide amongst yourselves what your working hours are as long as everything is done by the time that the MAG Chair says that it should be done, and then we can have a come together meeting as well when the MAG Chair says would be a good time to come together to give the final results of the final workshop selection process.

And we can also have the breakout rooms, Zoom has got breakout rooms.  We can use a Zoom breakout room or give them each of the streams different Zoom rooms.  So these are just ideas.  As I said at the beginning, if we have a mixture of face‑to‑face and virtual meeting, we do lose some of that flexibility because if we are having some people in the Palais, we have to stick to the Palais hours.  We cannot ask the staff at the Palais to work at 7:00 p.m., et cetera.  They have rather strict working times,  especially with the budget crunch that we do have.  At the UN no meeting happens after 6:00 p.m.  If we do have it virtually, the Secretariats are more than happy to work whenever they are needed to.  And we can be a little bit more flexible with our time.

So I will just stop there and hand it back to Anriette.  She can open up the floor or have questions or other discussion.  Thank you very much, Chair.

>> CHAIR:  Thanks, Chengetai.  Well, the discussion has been going on quite actively in the chat.  There is one just to focus on.  Veni made the point that if we are going to have an online meeting we need to consider making it longer than three days.  And I would definitely support that, and I think that's consistent with Chengetai's, the ideas that he shared about having different modules or different components of the meetings in different time zones.  That's one thing to comment on, the duration of the meeting.

And then there was also a proposal to postpone it.  And then there was a counter point made to that that if we postpone it we could run out of time.  But my sense from what people are saying in the chat is that we should not plan, that it would be unrealistic to plan for a face‑to‑face in June.  I think that's what I'm getting at the moment, but let me open the floor to more suggestions and comments on all of these proposals.

I don't see the queue.  Does anybody want to just jump in.  There is no one in the queue.

>> BEN WALLIS:  Anriette, it's Ben.  I can jump in quickly.

>> CHAIR:  Go ahead.

>> BEN WALLIS:  I very much agree face‑to‑face meetings are much preferred, but I don't think, I don't think there is much point in delaying it by months at this point there is so much uncertainty, I think, you know, we are still hoping that we Poland we can be face to face, but in terms of getting us to that place, I think we need to stick to our schedule which means doing the workshop evaluations in June.

So I think we need to find a way, and I appreciate all of the thinking that's gone into how we can do it and make it work, and I know I'm a good example of being on the west coast, but I'm certainly prepared to wake up early for a few mornings to help make it work.  Thanks.

>> CHAIR:  Thanks, again.  But I think as Adoma wrote again in the chat, it's a question of concentration span.  Spell find it very long all day long meetings.  They are demanding, it is difficult to be in a long meeting but we can look at structuring the agenda in a way that there are shorter chunks of work.  Especially if we use breakout groups.

I want to check one thing with Chengetai.  If we do have the June meeting as a virtual meeting, that does not mean we can't still have a face‑to‑face MAG meeting, am I correct?  We would have to focus on the different, we would work on the program on the main session, because we would have had to have completed the workshop evaluation, but would it be possible for us to have a face‑to‑face meeting later on in the cycle.  Chengetai, can you respond that that.

>> CHENGETAI MASANGO:  Yes, we can have a face‑to‑face meeting and we can have a face‑to‑face meeting as we intended sometime back with the WSIS Forum because that's postponed.  And if that goes on we can do that or we cannot.  But, yes, there is nothing stopping us from having a face to face meeting later on in the year.  People are talking about five days, so spending the online meeting for one working week, five days.  Is there support for that.

>> JUTTA CROLL:  I'm, its Jutta, may I step in.

>> CHAIR:  Yes, please, Jutta.

>> JUTTA CROLL:  First of all, I do think we need to consider when we will take the decision on the workshop proposals and there was already some comment in the chat that, of course, organizers of workshops need to know as early as possible that their workshops were selected for the program.  So if we are talking about postponing the decision on which workshop proposals go into the program we need to consider that thoroughly that we give information that a workshop is accepted out early enough for workshop organizers to have everything ready for the IGF in November.

I'm not sure whether postponing the face‑to‑face meeting when the decision is already taken earlier, because that might be necessary.  What would be the purpose of having additional face‑to‑face meeting later on maybe in July or August when, of course, also many people might have their summer holidays long awaited since there are no holidays in these days.

I'm not sure about that.  And finally, I do think it's will be better to spread the meeting if it's a virtual meeting over more than three days, but then we need a very good schedule because most people who do this, the MAG work is voluntary work beside their day‑to‑day jobs.  For me, it's just not possible to stay five days, a whole day or several hours in a MAG meeting because, of course, I have also other duties on my regular job.  And, therefore, I do think we need a very good structure when will be the meeting times for these virtual group meetings and then also we need a very good minutes or so not to repeat everything in the, when we come together for the next two hours and then for the next two hours and there are some people who have been in the first meeting, but not in the second meeting, but then in the third meeting.

So the flow of the debates on the program will be more difficult the longer we spread it.  So I'm not sure about how we get a very good structure to follow the meeting over such a period of time.  Thank you.

>> CHAIR:  Thank you, Jutta.  Roberto, you are next.

>> ROBERTO ZAMBRANA:  Thank you, Madame Chair.

What I think is really going to be difficult that we actually get rid of this crisis very soon, it's not going to be the same in all of our countries.  Some of the countries, perhaps, are going to start working sooner.  Some others later.  So I think it will be not a good idea to postpone the face‑to‑face, I mean, not the face‑to‑face meeting, the next meeting that was supposed to be face‑to‑face.  I think we should go ahead and think that this is going to be a virtual meeting.

And it could be good that we continue with our general time scheduled that we have so far.  What I will suggest is that perhaps in the following maybe days or weeks, we should try, I mean, the Secretariat could prepare a draft of an agenda considering all of these inputs that we are providing now in this way that it's going to be a virtual meeting.

I also agree with Jutta, it's not going to be easy for most of us to release our duties.  So I think we could think about the same, not the same time and structure, but at least the same number of days.  I mean, let's say we can have the three days considering that perhaps we are going to follow one of Chengetai's ideas regarding to receive some of the information, some of the presentations, record it.  I really don't think a good idea to have two different sessions, because most of the rich development on the meetings comes from the different points of view that we exchange between each other.

So I think it could be better to have just one session, but perhaps try to divide in different stage all of the presentations that can be recorded or can be prepared, even by groups, and keep some hours to distribute this kind of content, this kind of videos for all of the members to see.  A  slot of time during the day that most of the participants, most of the members will be able to attend, not given a conflict in times in their own sites.

If we if we manage to do it, it's probably that we are not going to use all of the eight hour daily, maybe not even five, maybe could be shorter, but that hours that we assign for the three, maybe four days that we assign finally to the meeting will be mostly dedicated to the exchange between each other, the interaction.  That's what I suggest.  Thank you.

>> CHAIR:  Thanks, Roberto.  Keep in mind, though, that if you were going to travel to Geneva for a three‑day meeting for most people outside of Europe in particular, that does mean a five‑day commitment, whereas now we would not have the travel time.  So it gives us and ultimately I don't think it's going to make a dramatic difference to the commitment we would have had to make to the meeting even if we spread it over more days, but I take both your and Jutta's points, Timea, you are next, and, yes, absolutely, Roberto, Chengetai, please note we should prepare a Draft Agenda for the next call.  Timea, you are next.

>> TIMEA SUTO:  I want to come in and support the ideas that Chengetai and colleagues were mentioning.  I do believe that come June, as pessimistic as I am, I'm sorry to say that I don't think the global situation will be either good to allow for all of us to travel or to have all of our minds completely of the fear of traveling to a large meeting or spending time in airports or anything like that.  I do think we should assume that this meeting is going to be fully virtual.

I also want to support those who are saying that a longer, more days could be added to this meeting.  More days doesn't necessarily mean more hours spent on this, so we could return to our other tasks in the morning or evening and just spend some hours doing more focused MAG and IGF work over that week, for example.

I want to reiterate a suggestion I made I think at the last MAG call, just coming from the practice that we had at the last face‑to‑face meeting of working in small groups really ad hoc small groups on certain ideas.  If we had more options for problem the Chair would ask to split into three, four groups based on how we were sitting in the room, and by a given time to come back with answers.

That's really easy to do online, and it's really easy to do in various time zones, and I want to stress here that time zones don't necessarily mean geographic or UN groupings.  Time zones in South Africa and in France are much more closer than east coast and west coast, for example.  So what I wanted to explore as well if we could find a time when plenary would be ideal, I guess that would be sometime around midday Geneva time that is the centre of the world as Chengetai said, try to focus plenary time together and try to identify those questions that we could break into smaller groups by time zone to it explore and come back to the Chair and the Secretariat with our agreed opinion as groups for the next plenary meeting.

And these groups can decide if they want to meet that afternoon or in the next morning if I'm using European time as a reference to come up with their suggestions and answers or questions to a problem before the next plenary.  This way we could move some of our ideas ahead, spread out the work but then also have less time per day spent on MAG and we can also focus on some of the duties that we all have in our day‑to‑day jobs.  Thank you.

>> CHAIR:  Thanks Timea, and thanks, everyone, for the great suggestions.  We have just under 15 minutes left, and I have Ben in the queue.  Anyone else?  Now is the time to join the queue, and if you have any other tunnels.

>> This is Mary.  I'm sorry, I cannot join the queue because I'm using the phone.

>> CHAIR:  That's fine, Mary, why don't you go ahead.  Ben, do you mind waiting for Mary?

>> BEN WALLIS:  No, of course not, thanks.  Go ahead Mary.

>> CHAIR:  Go ahead Mary.

>> Thank you.  My suggestion is very, very short.  My suggestion is we look at the hours when the world is to schedule the next things.  And I agree that we don't know when the whole world will be healed of this virus.  So the best thing is just to consider to plan the virtual meeting, and we will look at scheduling a plenary around the time that the whole world will be awake, east, west, north or south, that you will be able to join without having to be in the middle of the night or the wee hours of the day.  So the plenaries should be during those hours.

I don't know whether 12, the 12 noon to 2:00 p.m. UCC is very convenient for those that are in the Asia side region, so we should also look at it if we use the map of the world, we can see when people are aquake so we schedule in that time.  And we should have sent the number of these because as you said, Chair, if we have to travel, we have to spend at least five days to do a program.

But it will be very challenging for some of us, because WiFi connectivity, you will have noticed that I have been dropping off and coming back.  We have some challenges as well.  So for that, we factor those in, and then maybe we do our virtual calls will increase the number of virtual calls.  But let June just be concentrated for the workshop evaluation and agreeing on them.  If we can achieve that, then the rest will do when we do our virtual meetings.  Thank you.

>> CHAIR:  Thanks a lot, Mary.  Thanks for all of the useful, sensible input.  Ben, you are next.

>> BEN WALLIS:  Thank you.

Yes, indeed, what I was going to say follows a little bit what Mary closed with there.  One of the really critical decisions that need to be taken in the June meeting, if I remember from last year's June meeting, I thought it was very important that we got to the stage of deciding what all of the main session topics would be so that work could begin in late June, early July to start getting the main sessions moving.  That was a big improvement from 2018 when we didn't really start thinking about the main sessions until September.

So I think it would be worth using some time for the June meetings to decide how to deal with the main session.  But then clearly the other big decision is the decisions on the workshops.  And I just wonder especially as we are dealing with the need to do things more efficiently remotely and not face‑to‑face, if we can have a more quantitative than qualitative approach, and by that I mean that there are certainly some people who would want to or how can I say it. 

So if you just relied purely on the numbers and the rankings, that would tell you what the 60 workshops should be.  If you are trusting that the workshop evaluation system is, and you are trusting the numbers people are putting into the system, you can say we have had the MAG members review these and these were the top 60 workshops.

At one end of the scale, you can make that decision just based on how workshops did with the ranking.  And obviously you need to leave some space to be able to balance the program and possibly avoid some duplication, but the more that we can place trust in the rankings that are given and the less we try and open up into a discussion kind of to, you know, which denigrates the numbering, I'm not expressing myself very well today, but that would be one way in which we could make the whole decision making process in June easier by putting more store in the rankings that are received through the evaluation.  And it's possible that we can also get help from the Secretariat providing us with details in advance because of the amount of space and slots available and also with guidance from the Working Group on how the different thematic tracks should try and come up with their professional workshops in advance of the meeting.

So that's why not particularly well expressed on how we can focus on the main decisions in June and try to make it easier to come to a decision.  Thanks.

>> CHAIR:  Thanks Ben.

I think you were clear, and I think that's also, I think we can only achieve that level of certainly if all of the MAG members complete their scoring assignments.  I think we have not all MAG members complete scoring and view of workshop proposals, it comes easier to slide into reopening decisions.  I think you are points were clear and well taken.  So please, everyone, look at the chat.  There is useful comments in the chat.  There are some ICANN experiences being shared.

But I think we have now closed this Agenda Item.  We have covered it well and we have excellent ideas, lots of creative ideas.  I think we have a general sense of how to go about planning this meeting.  There seems to be consensus that we most definitely have this meeting as a virtual meeting around the existing dates.  That doesn't preclude that we might have a face‑to‑face meeting later on.  We can revisit that later, but the point now is we have to have this meeting to stick to our work schedule, and we will present the Draft Agenda to you on our next call for input.

Please continue to send ideas by email.  So now, to move onto the final Agenda Item, any other business.  Sylvia has put one item on the table, which is to ask for an update on next year's host.  Chengetai and I can share a little bit of information on that, nothing final, but there is some provisional information.  I see Wyman is also on the call so possibly he can add.  Any other items under any other business?  Is that the only one?  I don't see anything else.

>> CHENGETAI MASANGO:  For the host, we usually don't share the names of the potential hosts unless they give us the go ahead to.  Oh, but, I'm sorry, I will let you speak Anriette.

>> CHAIR:  I will, we don't, there is nothing final yet, but the Government did make a public announcement at an event in March that they have entered into discussion with New York about hosting in Adis next year.  It's not public information, discussions are going well, and maybe we can, maybe Secretariat can tell us more, but it's still a long way from being finalized.  But the indications are good, and I think the fact that they attended in Berlin.  They were very excited so that's a really positive step, and then in Adis we have two international institutions, there is UN Economic Commission and the African Union Commission which also has facilities.

So mending the outcome of this conversation between New York and Adis, we will probably know, I don't know, Chengetai when we will know, but the discussion seems to be proceeding well, but really, please do not consider this as public information that we can share.  It's just that it's on the table.

>> CHENGETAI MASANGO:  Anriette, we can't hear you anymore.

>> CHAIR:  Internet development Conference in March, the Deputy Minister.

>> CHENGETAI MASANGO:  You phased out.

>> CHAIR:  Am I back.  So which Chengetai pointed out that it was scheduled with ‑‑

>> CHENGETAI MASANGO:  I'm sorry, we can't hear you anymore.

  I think we may have lost Anriette.

>> CHAIR:  Hello.

>> CHENGETAI MASANGO:  Yes, hi, Mary, yes.

>> MARY ROSE RONTAL:  This is Mary.  Please in the meantime, can I ask where the November meeting is being considered to be held virtually, and if there is a toolkit that has been developed by IGF for the African IGF, we are also considering holding our 2020 IGF virtually, and we are looking for toolkit that are example or experience to be able to hold ours.  Thank you.

>> CHENGETAI MASANGO:  No.  For the November 2020 IGF meeting, we are still planning it.  Everything is going ahead exactly with no change that we are holding it in Katowice, and we are doing everything just like that.  I mean, it is November.  It may be a bit too far away to see what the exact situation on the ground is, but we are hopeful.  We are not putting on any breaks whatsoever as far as holding it in Katowice.

Now, that being said, we are observing, looking around, seeing what other people are doing, watching the situation, listening to WHO and also the host countries as well, seeing what they say about the situation.  And we are, because we are part of the UN family, and we are also part of the wider IG family as well.  We are helping and we are learning from other organisations who are holding their meetings virtually.  We are, you know, talking and exchanging views with, you know, people like UNCTAD, et cetera.

So that goes, so, I mean, that's also to answer your question about holding the African IGF virtually.  If you decide to do so, the, you know, the IGF Secretariat is here, Luis is here.  We have our Zoom licenses and our WebEx licenses and we will assist, you know, to the fullest of our ability, and also from what we have learned from all of the other people and what is going on that have held their meetings virtually as well.

Does that answer your question, Mary?

>> MARY ROSE RONTAL:  Yes, thank you very much.

>> CHAIR:  Anriette is back.  I'm sorry, my connection dropped.  It's raining here and that adds to that.

Jennifer asked when the decision will be made.  I think, as Chengetai said we are looking at the situation, but we will really take into account all of the constraints that we know people have to deal with, such as Visa application, such as travel arrangements.  So, you know, we would have to make, if we are going to not go ahead with an event, you know, we would have to give people a very long lead up time, three to four months.  So don't expect last‑minute decision.  We might not be able to have a final decision in June.  Hopefully we can, but we certainly can't leave it much beyond July.

Chengetai, everyone else, our time is up.  Anything else, any last inputs from anyone?  Can we close the meeting?

>> Anriette, it's me.

>> CHAIR:  Go ahead.

>> This is Wyman with UNDESA.

>> CHAIR:  Good to hear your voice and thanks for joining.

>> Thank you.  I just try to echo what you just said about the host country for 2021.

So you indicated is I have clear.  We understand that there could be some of this Twitter that were sent earlier, but following past practice, we certainly need to respect the decision of the future host country to make the announcement when it's ready.  And also given that this will be in Adis, there will also be a partnership with UN Economic Commission for Africa.  So this is a little complicated and we are having the discussion now.

If anything, the letters that the announcement will be ‑‑ latest at announcement will be made is at the Poland IGF.  We are not sure if the host country is ready to make the announcement earlier, but just to echo what you just said, just to keep the discussion within this level.  Thank you, Anriette.

>> CHAIR:  Thanks very much for that, Wyman.  It really, it is important, because hosting and host countries can be sensitive, it's sensitive information and it's important for us to keep a respectful and constructive relationship with all potential hosts.  So it is important that we handle this information with care.  But I think it is also good that we can look forward to an IGF that will be held in a developing country again.

So on that note, unless Chengetai is going to jump in with some more important issues, I want to thank everyone for joining the meeting.  Excellent attendance, I really appreciate it.  Thank you for your hard work.  More hard work to follow, but keep well, everyone, and look at yourselves and those that are close to you.  These are not easy circumstances.  So much uncertainty and different ways of living and working, but strength to all of you, and keep well and talk to you at the next call.  Chengetai, when will our next call be just to give everyone a heads up, please?

>> CHENGETAI MASANGO:  This time I'm actually ready if I can find where I wrote it down.  It's going to be on Tuesday, 28^th of April at 11:00 a.m.

>> CHAIR:  Okay.  Great.  Thanks very much for that.  And by then we will all be in the midst of workshop evaluation, I think.

>> CHENGETAI MASANGO:  Yes.

>> CHAIR:  Good.  And thanks, everyone, and goodbye, and keep well.  Thanks, Secretariat, thanks Wyman.

>> CHENGETAI MASANGO:  Thank you very much.

>> And stay inside!

>> Bye‑bye everyone.

>> Goodbye, everybody.